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FOREWORD 


This report is prepared in t^i'o volumes. Volume I reports the Executive 
Summary and tlie findings of die research. Volume n contains the appendices 
to die final report. The appendices list detailed documentation vhich supports 
the research findings. This includes specific materials and procedures 
used in: a) die open and closed forms of die knowledge tests, b) the full 
mission simulations, and c) the paper and pencil tests. 
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EXECUTIVE SUMMARY 


A critical In-fli^t event is a situation which is unexpected, unplanned 
and unanticipated, and is perceived by die pilot in command to direaten the 
safety of die aircraft. The CIFE is one which requires pilot Judgment beyond 
routine decision maldng or a pre-programmed decision structure and where 
die safety of the aircraft depends more on pilot cognitive processes than 
skilled motor performance. 

Research Objectives 

The objectives of this research were to: 

1) Describe and define the scope of the critical in-flight event with 
emphasis on pilot management of available resources . 

2) Develop detailed scenarios for both full mission simulation and 
paper and pencil (P/P) testing of pilot responses to CIFE's. 

3) Develop statistical relationships among pilot characteristics and 
observed responses to CIFE’s. 

These objectives grew out of a concern with anomalies in reported accidents 
and incidents in which some pilots or crews seemed better able to handle 
unusual in-flight events than others. For example, why did a professional 
crew piloting a Baltimore Colts 727 fail to recognize the symptoms of a 
frozen pitot system and subsequently enter a fatal stall-spin maneuver ? 

Contrast that event with the performance of an airline pilot who used differential 
power to overcome a locked elevator problem on his three-engine aircraft. 

Wltat characteristics of his training and decision making strategy permitted him 


ii 




to develop a successful solution to the problem ? Similar questions are 
raised by events such as a Cessna 206 pilot who experienced engine failure 
due to fuel exhaustion in one tank, and crashed the aircraft with the second 
tank nearly full of unused fuel. 

Project Development 

The project began with an early concern for the dynamics of CIFE 's and 
broad attempts to identify pertinent research issues. The final products 
were 1) a set of scenarios with associated hardware and techniques for 
studying CIFE phenomena in a basic general aviation fli^t simulator; 

2) a set of paper and pencil scenarios and associated techniques for stud}ring 
pilot diagnostic strategies and diversion decision making processes; 3) a set 
of testing instruments designed to measure a pilot's knowledge of aircraft 
subsystems and understanding of troubleshooting techniques; 4) a study 
relating cockpit crew procedural compliance widi performance errors . 
By-products of tibis research included one M.S. design project, one M.S. 
thesis, and a Fh.D. dissertation*. Major milestones in the project develop- 
ment are summarized in Figure 1. 

Model 

A five-phase model of pilot CIFE response is hypothesized on the basis 
of a) discussions with e.xperts in industry and government and b) observations 
made about pilot performance in both simulator and paper/pencil scenarios. 


♦See Appendix B 
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The five phases are: 


1) Detection 

2) Diagnosis 

3) Option Generation 

4) Decision Making 

5) Execution 

Information seeking activities permeate all five phases of this process. The 
inter-relationships and feedback among these phases are outlined in Figure 2. 

GAT -Scenar ios 

A Singer GAT-1 fli^t trainer was me iified to permit a variety of extra 
failure modes and to enhance data collection. Three scenarios were created 
to be tested in the GAT-1. These scenarios each involved a critical in-fli^t 
event imbedded in what was otherwise a routine simulated IFR flight. Subjects 
went through a pre-flight planning phase involving a complete weather briefing, 
route planning, and filing of fli^t plan. Take-off, climb and enroute phase of 
each scenario beg^n under normal IFR operating conditions. Real time ATC 
communications, including background conversations, were used to enhance 
realism. Some 20-30 minutes into each simulated fli^t one of the following 
critical events was introduced: 
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"Normal" 

Liniits 


HYPOTHESIZED PILOT CIFE RESPONSE 
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IDENTIFY OPTIONS 
SOP 

a/c System Aids 
Manual Aids 
ATC Assistance 
Creative Experiments 
Analog-ies From Experience 




I SELECT OPTION 
(Match Goals 


Estimate Outcome 
Match Capabilities 




EXECUTE OPTION 


Unsatisfactory 


EVALUATE PROGRESS 


Monitor Systems 
Compare States 


satisfactory 


COMPLETE EXECUTION 


HYPOTHESIS: 5 Phase Model . Inf ormatio 
Seeking Permeates All Phases. 

A. DETECTION: Items 1 thru 5 

B. DIAGNOSIS: Items 6 thru 8 

C. OPTION GENERATION: Item 9 

D. DECISION MAKING: Items 10 thru 1 

E. EXECUTION: Item 13 


Fig^e 2 





1) Fuel starvation on the active tank (as might be encountered 
because of a loose fuel cap). 

2) Partial power failure (as might be caused by a broken baffle in 
a muffler). 

3) Navaid loss (as mi^t be caused by failure of a single airborne 
receiver component). 

Sidbject performance was observed through one-way windows on the 
simulator and recorded by video tape, a 3-channel audio tape and written 
evaluations by the three ejqjerimenters present. These data were later used 
to measure "stick and rudder" skills and communications techniques as well 
as to map each pilot's response to the critical in-flight event. 

Twelve subjects were selected for testing in the full mission GAT scenarios. 
Althou^ all were IFR rated, they ranged in age from 20 to 56 years old, in 
fli^t e^qperience from 270 to 8800 hours and in certification from private pilot 
to ATP. Each subject was given two different forms of knowledge survey to 
complete and was thorou^ly debriefed after his flight. 

A wide range of cockpit management styles and apparent skill levels were 
observed. Although it was difficult to quantify, "good performance" was 
easily recognized by the observers of the experiment. The elements of "good 
performance" included: 

1) professional use of the radio 

2) precise headii^ and altitude control prior to and during the CIFE 

3) constant awarenes of the aircraft position along its intended route 
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4) prompt, but not necessarily instant, response to the on-set of the 
CIFE (detection) 

5) systematic procedure for trouble shooting 

6) knowledge and use of available ATC resources 

7) diversion decisions which allowed for further potential uncertainties 
The sample was too small to indicate anything other than some initial 

h 3 rpotheses concerning pilot performance in such a full- miss ion setting. 
However, the following tendencies were noted: 

1) Cockpit management style varies widely among pilots. For 
example, some are extremely self-reliant, others want 
immediate and extensive help from ATC while still others 
make the decision making process a joint effort with ATC. 

2) Good stick and rudder pilots seem to have excess capability 
and maintain good stick and rudder performance during and 
after the CIFE. More marginal stick and rudder pilots, on 
the other hand, show increased frequency and amplitude of 
heading and altitude excursions, and e:qierience commimication 
difficulties when faced with a CIFE . 

3) Pilots who score well on the knowledge test instruments tend 
to perform well in problem diagnosis and decision making. 

From the observations of the experimenters and comments made by 
participating subjects, it appears that such a full mission simulation exercise, 
coupled with an appropriate knowledge survey and debriefing, could be a 
valuable tool for recurrent training of IFR pilots. 


viil 



Paper and PeacU Scenarios 

Paper and pencil (P/P) scenarios, and associated experimental techniques, 
were created to streamline the data collection and analysis for pilot responses 
to critical in-fli^t events. Although they lacked the high stress environment 
of the GAT-1 experiments, these scenarios did yield useful data on pilot 
problem diagnosis and decision making skills and strategies. 

The paper and pencil scenarios have the following advantages over the 
GAT-1 scenarios; 

1) Experimental conditions are more easily replicated between subjects. 

2) Data collection is more easily accomplished. 

3) Diagnostic capabilities and decision making strategies can be more 
easily isolated. 

4) They are much cheaper, in terms of both time and money, which 
means that a much larger sample size of subjects can be run. 

The paper and pencil experiments were conducted in a workshop- type 
environment. A group of std>jects, usually Jiree or four, were seated in a 
conference room for a common briefing and initial testing. Each subject was 
asked to complete a background questionnaire, which asked for data on his 
personal flying experience. Items such as age, ratings, total flyiPb time, 
recency of experience and type of flying most often done were included. The}' 
then were given a 20-question knowledge survey (multiple-choice questions) 
designed to measure their knowledge of aircraft sub-systems and trouble- 
shooting skills. After the tests were completed, the group was given a complete 
briefing on the equipment to be flown, the weather expected, and the airspace 
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in which they would be assumed to be operating for purposes of their CIFE 
scenarios. At that point the group was disbanded with each subject accompany- 
ing a single experimenter to a private room where the scenarios were administe-‘“d. 

Two sets of scenarios were used on each subject. The first set consist 
of four scenarios directed toward problem diagnosis. The second set 
involved two exercises designed to explore diversion-decision making strat- 
egies of pilots. (A diversion decision involves choosing an alternate airport 
when the intended destination airport is unavailable due to a CIFE . At the 
completion of the paper and pencil experiments the subjects were invited to 
tour the GAT-1 simulator used in the earlier study and to participate in an 
informal debriefing. The entire process required about ninety minutes from 
beginning to end. 

For these tests, forty subjects were used. Almost all were current 
instrument-rated pilots with ages ranging from twenty to sixty- five years, with 
both civil and military backgrounds, and embracing total flying experience from 
270 to 19,000 hours. As a group, these pilots were considered to be above 
average in experience. 

p/p Diagnostic Scenarios 

Four separate diagnostic problem situations were presented to each subject. 
Thet3 scenarios centered about problems presumed to be created by: 

1) an oil leak at the oil-pressure gauge line 

2) a vacuum pump failure 

3) a right magneto drive gear failure 

4) a frozen static pn»‘t 
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After Instructions for the diagnosis scenarios were read to the subject, 
he was given an aircraft instrument panel layout diagram and enroute chart 
for the first problem. The scenario was then read, concluding with a 
statement of a major symptom, e.g. , "You smell hot engine oil. What 
would you do?". The subject was given a maximum of four minutes to seek 
information from the experimenter and conclude his diagnosis of the problem. 
He could ask for any information available from instruments noted on .is 
panel diagram, response to control inputs or external cues such as oil on 
the windshield or ice on the wings. The experimenter had a diagnosis infor- 
mation checklist from which he provided information in response to the 
subject's request. For example, if the subject asked for oil temperature, 
the experimenter would respond "normal" if that was the entry on his check- 
list. As each piece of data was requested, its order was noted on the exper- 
imenter’s checklist. If a diagnosis was not offered by the subject prior to 
the elapsed time (four minutes) the subject was asked for his best estimate 
of the diagnosis at that time. At the completion of the allotted time the subject 
was asked to estimate the criticality (scale of 1 to 7) of the problem as he 
perceived it. Then he was given the correct diagnosis and was asked to 
re-estimate the criticality in the light of this perfect information. The same 
procedure was repeated for each of the four scenarios. 

Eight pieces of basic information were extracted from each diagnosis 
summary sheet. These were: 
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1) Number of inquiries. (An inquiry represents a request for 
a single piece of information . ) 

2) Total tracks of inquiries. (A track represents a single coherent 
line of questioning which may involxs several inquiries; for example, 
fuel pressure, fuel flow, fuel gauge status.) 

3) Unique tracks of inquiries. (A ^lUjject may start one track, 
abandon it, shift to a second track and then return to the first 
track. Althou^ three total tracks would be noted, only two 
unique tracks exist. ) 

4) Correctness score. (A score of 1 to 5 was given which reflected 
how close the subject's final diagnosis was to the "perfect" 
solution. ) 

5) Time to complete the diagnosis. 

6) Criticality estimate before the correct diagnosis was revealed. 

This was a subjective-rating scale, 1-7. 

7) Criticality ^ timate after the correct diagnosis was revealed. 

8) Number of control input inquiries. (A control inquiry Involves 
movement of an aircraft control, e.g. , "What happens if I 
advance the throttle?") 
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These primary data were then used to create a number of compound 
performance measures including estimates of efficiency and merit. 

"Efficiency" was measured by the time and number of inquiries required 
to reach a diagnosis. Subjects «dio reached their diagnosis quickly (be it 
right or wrong) and who made relatively few inquiries received high eiticiency 
scores. "Merit" was measured by multipl3ring correctness and efficiency 
scores on a given scenario. 

p/p Decision Making Scenario 

The decision making phase of the paper and pencil experiments was 
divided into two parts, an information seekii^ part and a rank ordering 
of alternatives. The basic scenario used for both phases involved a hypo- 
thetical flight in a Cherokee Arrow from Bangor, Maine to Glens Falls, 

New York for a business meeting. Weather along the route and at the 
destination was marginal with rain, low ceilings and drizzle signifying 
instrument meteorological conditions (IMC). The scenario was read to 
the subject as he was invited to follow the progress of the hypothetical fli^t 
along an enroute chart. About midway along the route the aircraft encountered 
an alternator failure, the diagnosis for which was clearly defined for the 
subject. .An upper limit on the length of time battery power alone would 
run the required electrical equipment was then given. This maximu' time 
(exact time was uncertain) was less than the time required to reach the 
primary destination, thus forcing a diversion decision on the pilot. 
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For U.C informatiOD seeking task the pilot was supplied with a simplified 
enroute chart with sixteen airports indicated by letters along his flight patib. 

The subject was then givvn two minutes to ask for information about any of 
diose airports. For each airport questioned, there were six pieces of 
information the experimenter was prepared to provide: 

1) Bearing and distance from his present location. 

2) Ceiling at the airport. 

3) Visibility at the airport. 

4) Approach aids available. 

5) ATC services a\'ailable. 

6) Terrain surmunding the airport. 

The experimenter provided the pilot with each piece of information requested and 
the e.xperimenter recorded the sequence in which it was requested. The pilot 
continued to request information until he had selected an airport (or until 
forced to seleci at the end of two minutes) and revealed his choice to the 
experimenter. 

For the ranking of alternatives phase, the pilot was asked to rank each 
of sLxteen alternative airports. He was provided with ATC facilities, ceiling 
and visibility, time to be reached and approach aids information on all airports. 
The airports were to be ranked from "most preferable" to 'least preferable" 
given his problem situation. No time limit was imposed for this task. In 
order to assess his risk-taking tendencies, the experimenter posed a series 
of questions for the pilot to consider after he had obtained his ranking. The 
questions asked how far he would go down his list of ranked airports to find 
one with maintenance facilities to repair his airplane. 



Hie data from the ranking task were used to determine Uie relative worth 
structure. The coefficients, or weights, for die variables ATC, weadier, 
time, and approach were obtained by regression anal]rsis according to die 
techniques of conjoint measurement. The range of values for the coefficients 
was 0.250 to 4.000. The relative wordi coefficients were later used to 
determine if any relationship existed with pilot background variables, results 
of the knowledge survey, diagnostic ability, and search pattern exhibited in 
the information seeking task. 

p/p Scenario Generalizations 

For purposes of analysis the closed form (multiple choice) knowledge survey 
was considered to be part of the P/p experiments. This knowledge survey 
focused on aircraft subsystems and trouble shooting in three major areas; 

1) engine and biel systems, 2) electrical systems and cockpit instrumentation, 
and 3) weather and IFR operations. 

A series of Spearman Rank Correlation studies, stepwise regression 
analyses and t-tests were performed on the combination of pilot background 
variables, knowledge survey results, diagnostic scenario performance and 
decision making measures. Among the observations made from these analyses 
are the following: 

1) There is no correlation between knowledge score and total 
fli^t hours. 

2) Knowledge score is correlated with pilot ratings held. 

3) Pilots good in one section of the knowledge survey tend to be 
good in all sections. 
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4) Diagnostic performance is hi^ly correlated 'vith knowledge scores. 

5) Knowledge is inversely related to total diagnostic inquiries, 
e.g. , knowledgeable pilots reach conclusions drifi^t or wrong) 
more rapidly dian others. 

6) Total diagnostic inquiries is inversely related to correctness, 
this sugg 'ts that undirected experimentation is poor diagnosis 
style. 

7) Total diagnosis correctness score is correlated uith efficienc}'. 

8) Civil trained pilots place a hi^er worth on ATC service in 
diversion decisions than do military pilots. 

9) Private pilots place a higher worth on weather factors in 
diversion decisions than do commercial and AT^ rated pilots. 

10) ATP rated pilots place high worth on time in diversion derisions. 

11) Pilots with good diagnostic scores place less wei^t on approach 
aids in diversion decisions. 

12) Pilots with good diagnostic scores place more weight on time in 
diversion decisions. 

13) The pilots with good diagnostic performance were characterized 
as knowledgeable about aircraft systems, employed few tracks 

to get at an answer, used few inquiries per track, and emphasized 
time in their destination diversion decision. They were not 
differentiated by flight hours, ratings, training, or t>pe of flying. 
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Procedural Compliance 

In support oi die general research objectives, Sdiofleld investigated 
airline cockpit crew operations. For his dissertation he used data generated 
in an experiment conducted in 1976 by Dr. H. P. Ruffell Smith under die 
auspices of die NASA-Ames Research Center*. Ruffell Smith used a full 
mission simulation scenario of a Boeing 747 flight to study crew errors 
generated during hi^ workload segments of the simulated flight. Schofield 
used the same data to study routine tasks of Qi^t operations during low work- 
load segments of that flight. He was concerned with: 

1) Quantifying routine procedures. 

2) Analyzing observed crew errors to identify which particular crew- 
members were the primary causes of such errors. 

3) Comparing measures of procedural compliance and operator error. 

Schofield identified nineteen separate words and phrases associated widi 

airc w operations which had procedural connotation. Using that list as the 
basis for definition he enumerated 97 normal oparating procedures which 
eculd be identified as standard cockpit activities in a 75-minute fli^t. This 
list did not include any abnormal, alternate, irregular or emergency pro- 
ced'ires. 

Twenfy-one crew coordination procedures were separated from the total 
list for further study. This group was emphasized because those procedures 
caqptured t^e essential ingredients of group leadership, cr?w management, 

*See Appendix B 
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and behavioral conformity. Schofield sought to examine relationships between 
meticulous complia«H;e with coordination procedures and the crew errors 
noted by Ruffie Smith. 

Schofield selected ten runs, which had the same set of observers and 
usable audio data throughout, for detailed procedural analysis. The 21 crew- 
coordination procedures were further subdivided into check lists, call outs, 
configuration changes and transfers with each of the ten crews evaluated in 
each subdivision. 

The prescribed command-announcement-challenge sequence for check- 
list procedures was fully executed in only five of fifty opportunities, when die 
crew members involved were pilot and co-pilot. When the flight engineer 
was involved, fifteen of thirty opportunities were fully executed. Schofield 
h}rpothesized that crew coordination might be improved by making the fli|^t 
engineer the challenger of all checklists. 

One hundred seventy opportunities, among the ten crews, to execute 
callout procedures were noted. Thirty ei^t procedural errors were 
identified, half of which were errors in altitude callouts during climb or 
descent. 

The 104 observed configuration changes, e.g. , gear and flap extensions, 
were well executed in terms of established oral procedures. Verbal 
indicators of transfer of EGT monitors were also given with few omissions. 
However, the optional transfer of control procedure was seldom observed 
even though opportunities existed to use it. 
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Schofield used stepwise multiple regression techniques to identify the 
best models relating die independent (procedural) variables to each of the 
dependent (error) variables in turn. He found that dependent variables 
which reflect errors by the fl 3 ring pilot, by the captain, and by the two 
pilots collectively all have hi^ly significant regression models in which 
pilot flying checklist commands and non-flying pilot callouts are the common 
independent variables. 

The Schofield study of procedural compliance by aircrews who participated 
in the Ruffell Smith eiqperiment suggests die following observations: 

1) Crew members face an impossible challenge in attempting to 
mentally catalog all of the standard operating procedures (SOP) 
pubiished for them. 

2) Routine non-compliance with an assortment of SOP's has been 
documented. 

3) Human redundancy by itself does not er radicate personnel 
errors. 

4) A statistical link appears lo exist between operator errors 
and procedural compliance. 
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I. INTRODUCTION 


Departure: 75 Y what Is your altitude? 

N1675Y: Columbus Departure Control, what do you show our altitude on 
our encoding altimeter ? 

Departure: I show you at 500 feet. That's why I asked you. 

N1675Y: It shows 1000. 

Departure: OK. Stop altitude squawk. I show you at 400 feet new. 

Obviously it's not working right. 

N1675Y: 75Y we're having problems with airspeed and everjthing here — 
What do you show our airspeed ? 

Departure: OK. 7 5Y do you want vectors back into the airport ? 

N1675Y: Yeah! Let's do that. 

The above brief excerpt of an actual communication between .4TC and a pilot 
experiencing in-flight problems in IFR conditions typifies a persistent dilemma 
in aviation . ^ We do not (nor does the air traffic controller ) understand the true 
nature of this pilot's problem. How long has he experienced airspeed and altitude 
problems ? Is it a matter of structural ice, mechanical failure or pilot error? 
If an emergency is to be declared, what does the fact of declaring an emergency 
mean to the air traffic controller? WTia*’ does it mean to the pilot? (A confession 
of Incompetence — an invitation to loss of license? Is there a r.eed to specify 
intentions ? Can the pilot provide intentions if he is unaware of the options 
open to him? How can we avoid those situations in which the pilot relinquishes 
command to someone on the ground? 

^These excerpts from a communications tape are verbatim. Only the air- 
craft identification number has been changed. 
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The assessment of the criticality of the situation demands more information. 


such as the pilot's capability, his training level, his experience with in-fl'ght 
problems, weather, location, terrain, altitude, etc. Yet this situation is but 
an example of many such events that occur each year in our national aviation 
system. 

Each year air traffic coniro' rovides several thousand assists to pilots. 
In 1970, of the 4, 187 assists. 53 < involved lost pilots, but 25 % involvv..! 
fuel problems, navigational failures, and mechanical problems. How many 
problems w'ent unannounced and resulted in tragic consequences for lack of 
pilot understanding of how to cope with in-flight problems ? How many 
emergencies were declared which couid have been avoided and reduced dis- 
ruption in air traffic control systems ? 

Discussions with pilots of various experience levels and ratii^s reveal 
little agreement as to when to declare an emergency and the operational and 
legal consequences of such a declaration. There are instances wherein pilots 
have risked and lost their lives and those of the passengers to avoid possible 
suspension of license as a consequence of declaring an emergency when they 
believed they had violated a regulation. (See NTSB-.4AR-71-1). Do the 
perceptions of the air traffic control personnel differ from that of pilots in 
this regard? Most importantly, can pilots be trained to handle in-flight 
problems, provide early assessment and intelligent response to the situation? 
What should a pilot do if: 

a) strange noises occur ? 


b) the door opens in fli^t? 



c) the ammeter shows a discharge? 

d) the radios fail? 

e) smoke aj^ars in the cockpit? 

f ) he cannot determine his location in his flight progress ? 

g) the weather closes in on him ? 

Some empirical evidence of pilot perception of threatening experiences is 
shown in Tables 1 and 2. These data suggest that a pilot's ratings, type of 
operation . and implied skill level, all serve to alter his perception of critical 
in-fli^t events. 

What should the "system" be able to do to assist the pilot in properly 
assessing his real (or perceived) problem? No simple answer exists for 
these situations nor does past research appear to address these issues. It 
is hard to imagine the extent of myths and misconceptions about critical in- 
flight events. Critical events lead to air traffic control disruption, panic, 
accidents, and perhaps firm resolutions by pilots never to fly again. 

In the NASA Aviation Safety Reporting System, 1,497 Incidents were 
submitted in the period of July 15 to October 15, 1976. Of these 3% involved 
aircraft structure and subsystem factors and about 9% navigation and communi- 
cation situations. If one assumes that the Reporting System captures only a 
portion of the total incidents occurring in the system, this statistic also 
suggests there may be thousands of critical in-flight events each year. 

While one objective of this research is to describe and define the scope of 
the critical in-flight event (CIFE), a definition or set of qualifiers for the 
purposes of this report is set forth below. 
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i-i. Reported Most Uncomfortable or Threatening Experience During An IFR Flight 
In Actual IFR Conditions (Reproduced from Study to Determine the Operational 
Profile and Mission of the Certificated Instrument Rated Private and Commercial 
Pilot, FAA-RD-70-51 July 1970, p. 125) 
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* Total « 739 

** Total » 1767 



Tabl« 1-2. Most Threatening Experiences Reported by Private 
Pilots (R^roduced from Study to Determine the 
Flight Profile and Missi<m of the Certificated 
Private Pilot, FAA-DS-68-1S, July 1968, pp. 81-82.) 


Reports 


_ Threat enino Experience 

Number 

% of Total 

(1) 

(2) 

(3) 

Lovr visibility 

338 

29( 

Crosswind 

290 

24 

Low celling 

277 

23 

Malfunctions 

254 

21 

Landings 

247 

21 

High winds 

229 

19 

In fog or clouds 

209 

18 

Near air collision 

198 

17 

Lost 

148 

12 

Short field 

147 

12 

'Fuel supply 

139 

12 

Engine operations 

122 

10 

Forced landing 

117 

10 

TaXeoffs 

111 

9 

Unimproved airport 

96 

8 

Mud or snow 

96 

8 

Darkness 

92 

8 

Weight or loading 

88 

7 

Infreq[uent piloting 

87 

7 

Trees or wires 

71 

6 

Use of radio 

64 

5 

Soft field and high grass 

64 

5 

Different type aircraft 

42 

4 

Pre flight operation 

38 

3 

Unavailable preflight information 

3S 

3 


IL 
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Table 1-2 (Continued) 


Reports 


Threatenino Experience 

Humber 

% of Total 1/ 

(1) 

(2) 

(3) 

Improper airspeed 

30 

3 

Stalls or recoveries 

28 

2 

Low altitude maneuvering 

26 

2 

Uninformed 

23 

2 

Steep turns 

21 

2 

Flaps 

20 

2 

Handling of aircraft 

19 

2 

Holding altitude 

18 

2 

Check list 

14 

1 

Slow speed flight 

13 

1 

Flight materials (maps, etc.) 

13 

1 

Pontoons or skis 

11 

1 

Slips 

10 

1 

Other 

136 

11 


1/ Total = 1,192 
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A critical in-fli^t event is a situation which is unexpected, 
unplanned, and unanticipated, and is perceived by the pilot 
in command to threaten the safety of the aircraft. The CIFE 
is one which requires pilot judgpnent beyond routine decision 
making or pre-programmed decision structure. It may or 
may not involve communication with ATC. The CIFE assumes 
alternative courses of action are open to the pilot and some 
finite period of time is available to the pilot to make an assess- 
ment of the situation, enumerate options and make a d jision. 

The safety of the aircraft depends more on pilot cognitive pro- 
cesses than skilled motor performance. 

For purposes of this research, emphasis was placed on IFR rated pilots who 
have sufficient experience to utilize the ATC system when available. 

Many examples of the above description can be put forth. The following 
illustrates a few of these. 

a) failure of navigational equipment, 

b) failure of electrical systems, 

c) failure of hydraulic systems, 

d) fuel management problems, 

e) flights into unexpected weather, 

f) unforecast icing conditions, 

g) engine failure (single and multiengine aircraft), and 

h) partial pilot incapacitation. 

This research was directed towards an understanding of: 

a) the nature of critical in-flight events (CIFE), their causes, and 
how they develop over time; 

b) how pilots of different backgrounds might assess and respond to 
such instances; 
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c) the psychological stress of in-flight events, appropriate coping 
processes, and the modeling of such processes; 

d) the interaction that exists between air traffic controllers and 
pilots during CIFE's; and 

e) how adequate countermeasures can b** developed from the above 
to minimize the frequency and consequences of CIFE's. 

An explicit description of research objectives and discussion of the scope 
of the project are presented in the next section. 
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n. RESEARCH OBJECTIVES AND PROJECT SCOPE 


The general objectives of the research were: 

1. To describe and define the scope of the critical in-fli^t event with 
emphasis on characterizing 

a) event development, 

b) event detection, 

c) event assessment, 

d) pilot information requirements, sources, acquisition, and 
interpretation, 

e) pilot response options, 

f ) pilot decision processes, 

g) decision implementation, and 

h) event outcome. 

2. To develop detailed scenarios from (1) above for use in 

a) simulators as well as paper and pencil testing for developing 
relationships between pilot performance and background information, 
and 

b) an analysis of pilot reaction, decision, and feedback processes . 

The scen<irios are viewed as data generating devices for pilot options. 
More specific thrusts of this research, related to the general objectives 
above, were developed on the basis of initial research findings and research 
capabilities. These Involved: 

a) emphasis on general aviation IFR pilots In single engine aircraft 
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b) emphasis on the descriptive character of pilot response to 
critical in-flight events 

c) use of full mission simulation 

d) use of paper and pencil scenarios to study pilot problem diagnostic 
capabilities and destination-diversion decision processes 

e) exploration of the relationship between procedural compliance 
and flight crew errors using the Ruffell-Smith simulation data. 

The following chapters place major emphasis on: 

1) background acti\itles leading to problem conceptualization 
f Chapter HI) 

2) development of knowledge tests on system anomalies (Chapter IV) 

3) full mission simulation (Chapter V) 

4) paper and pencil seen rio tests (Chapter VI), and 

5) analysis of the Ruffell-Smith data for procedural compliance 
(Chapter VII) 
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m BACKGROUND 


Literature Review 

Initial project activities centered around the development and implementa- 
tion of a comprehensive literature search. Because the objectives of the 
project were rather broad ranged and cross-disciplinary, this search involved 
a number of topic areas. After an extensive review of search materials 
available a master list of kej'^ words to be used in all literature searches 
was developed. This list was used for all searches with the exception of 
pscyhology abstracts which used a controlled vocabulary. This controlled 
vocabulary can be found in The Searches of Psychological Index Terms , pub- 
lished by the American Psychological Association. The following sources were 
examined: 

a) The Cfclo State University Mechanized Information Center (C6U-MIC) 

b) Psychology Abstracts 

c) FA A Accident Reports 

d) Transportation Research Information System (TRES) 

e) National Technical Information Service (NTIS) 

f ) Department of Defense sources (see Appendix A) 

g) Aviation Press Publications, e.g. , Flying Magazine, Business and 
Commercial Aviation etc. 

h) National Transportation Safety Board (NTSB) Accident fleoorts 
While the literature was replete with "never again" stories, surprisingly 

few documents addressed pilot response to critical in-flight events in 
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sufficient detail to permit pilot response modelling. Indeed, little 
statistical evidence was available on the relative frequency of various types 
of incidents. 

Appendix B is an annotated bibliography of some of the literature examined. 
In addition, the dissertation by Schofield and the thesis by Flathers detail 
further background sources in this area. 

B. Results of Interviews With Interested Agencies 

At the outset of this project the principal investigators met with several 
organizations which had both a vital interest in the problem and expertise in 
pilot behavior. The National Transportation Safety Board (NTSB), The 
Aircraft Owners ana Pilots Association (AOPA), Mitre Corporation, Airline 
Pilots Association (ALPA), Air Transport Association (ATA), National 
Aeronautics and Space Administration (NASA), Federal Aviation Administration 
(FAA), Air Force Office of Scientific Research (AFOSR), and United Airlines 
(UAL) were all visited to provide consultation with their staffs on their per- 
ceptions of the CIFE and to secure whatever data bases were available to 
document the extent and nature of CIFE and any data on related pilot response. 
These agencies also suggested other resources for this problem area - either 
published reports , research in progress or names of individuals who could 
provide insight into the CIFE problem. Trip summaries and contacts are 
outline 1 in Appendices B and C. 

In general, all agencies reported a great interest in the problem and were 
willing to help within their capacities but admitted that the CIFE was largely 
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an unresearched issue. No data bases on pilot response to CIFE's were 
available. There were, to be sure, many shared experiences, individual 
examples of CIFE's from FA A and NTSB files and unique perceptions from 
those Interviewed. For general aviation (GA) CIFEs there were little or no 
data available from NTSB, FAA or NASA-'ASRS files. From discussions 
with these agencies and among the research staff, several hypotheses or 
constructs were proposed about the CIFE process such as 

1) response latency theory 

2) social interaction in the cockpit 

3) cognitive structuring 

4) pilot workload 

5) detection of vs. response to CIFE 

6) appraisal of CIFE's 

7) single channel limitation of the pilot 

8) lack of standard work habits 

9) lack of real world elements in training and testing of pilots 

C. Results of the NASA-ASRS Search 

Eany in the research, the project team asked NASA-Ames to perform 
a search of its ASRS data file. Using key words consistent with their data 
base structure, e.g., emergency, pilot decision making, etc., some two 
dozen narratives were developed and examined, in general, little value to 
the project resulted from this search principally because of lack of detail 
about how the problem developed, how it was diagnosed, what alternatives 
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were considered and other relevau» details, e.g. , weather and alternate 
airports available. Because of (he NASA policy on anonymity of the reporter it was 


impossible to trace back an incident to get more information. WTjat would 
have been necessary would have been to have the analyst follow up immediately 
on first contact using a supplementary data sheet. Because the thrust of the 
research was directed towards the GA pilot and the fact that the majority of 
incidents reported were air carriers, it was decided not to pursue the 
ASRS data file further. 

D. An Initial Conceptual Model 

As a result of background information, discussions with experts and a 
graduate seminar directed to the various facets of the problem, a preliminary 
model of this process evolved. This is showTi in Figure III-l. This conceptual- 
ization depicts several key aspects of the problem - the detection phase, 
information seeking strategies, workload, use of resources, and pilot 
stress, decision styles and value systems making up his decision making. 
Ultimately, pUot response was focused upon: 

1) detection of the problem 

2) diagnosis of the cause from the symptoms 

3) generation of viable options 

4) decision making both in terms of problem resolution and 
destination diversion 

5) execution of the decision 

Throu^out all five phases, pilot information seeking strategies were studied. 
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A MODEL OF RESPONSE TO A CIFE 


Pilot Factors 
(Independent Variables) 


Situational Factors 
(Dependent Variable = *) 



Figure III-l 
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E. Research Strategy 

Figure HI- 2 depicts the overall strategy undertaken in this research. 
Above the dotted line are the initial background efforts. These included die 
aforementioned visitations, literature review, ASRS search and graduate 
seminar. Prior to the formal research initiation, a graduate project by 
Fox, "Critical In-Flight Responses", indicated the potential value of using 
paper and pencil scenarios to study pilot decision making. At the same time, 
as part of another Industrial and Systems Engineering (ISE) course, USAF 
pilots were surveyed to arrive at candidate scenarios for future simulation 
or paper and pencil testing. Both of these exercises provided encouraging 
results. 

Below the dashed line are the four major fronts of the project: 

1) the development of knowledge tests 

2) full mission G.AT simulations 

3) paper and pencil scenario testing 

4) the relation of procedural compliance to errors in the Ruffell 
Smith simulation study 

Each of these major fronts is discussed in turn in the chapters which follow. 

A comment on the paper and pencil scenario workshop is in order. This 
was a mechanism to bring pilots together for a general briefing prior to 
their testing on paper and pencil scenarios. These tests were then conducted 
with individual experimenters. Hence, about ten small group workshops were 
held on different days. 
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IV. EVALUATION OF SUBJECT KNOWLEDGE 
OF AIRCRAFT SYSTEM ANOMAUES 

Tile evaluation of subject knowledge of aircraft systems and tiUe IFR 
operating environment was an important element in understanding the ways 
in which pilots respond to CIFE's. The evolution of the final test instrument 
involved a series of pre-tests, development of open form questions and finally 
a closed-form, multiple-choice questionnaire. Contrary to the usual airman 
certification exam question format, the bulk of the items selected here 
emphasized aircraft subsystem operation and trouble-shooting. 

A. Open Form Survey 

A knowledge survey, or inventory, was developed to determine the level 
of a pilot's knowledge of aircraft systems and the IFR operating environment. 

An open-form survey was administered to pilots who were participants in the 
GAT runs. Later a closed- form version was administered to all subjects in 
the paper and pencil scenarios. The results of the survey’s were compared to 
various measures of pilot performance in the simulations to isolate relation- 
ships between pilot knowledge level and measureable aspects of piloting skill. 
The development of the survey followed a three stage process which included 
1) item (question selection and pre-test, 2) construction and test of an open-iorm 
survey, and 3) construction and test of a closed-form (multiple choice) exam. 

The items for the survey were constructed from information in training 
texts, government publications, aircraft operating manuals, and other 
readily available publications. Practice quizzes and examina- 
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tions commonly used in die certification process placed much emphasis on 
such areas as regulations, weather, navigation, and wei^t and balance. 
Relatively few items on aircraft subsystem operation or trouble-shooting 
could be found. This is due, in part, to the fact that a modern single engine 
aircraft may have over 25 independent systems , and some of these systems 
may be engineered differently by the various manufacturers. For example, 
while the pitot-static sjrstem and gyro- instrument system designs offered 
by aircraft manufacturers are fairly uniform, other systems such as fuel 
metering and feed devices are often vastly different. Special care was taken 
to ensure that items selected for the knowledge survey were representative 
of the types of systems pilots could be reasonably expected to encounter in 
their flying careers. 

A total of over 60 items were collected and pre-tested on a small group of 
pilots. Included in this prototype survey were areas such as fuel systems, 
electrical systems, engine systems and operations, cockpit instrumentation, 
weather and the flight environment, and general IFR procedures. The pre-test 
survey items were presented in the form of open-ended questions to which the 
pre-test subjects responded with short, written answers. Four tsrpes of 
questions were posed. The first t}rpe was a simple, straightforward question 
in which the pilot was asked to define or explain something. In the second type 
of question, given certain symptoms in terms of instrument indications, noises, 
visual inspection, and the like, the pilot was asked to identify the most likely 
cause of those symptoms (Symptom-Cause, or S-C). In the third t^qie of 
question, the pilot was given a specific condition and was asked to identify 


19 



symptoms that would most likely arise from ttiat condition (Cause-Symptom, or 
C-S) . In the fourth and final type of question, flie pilot was asked what 
corrective action should be taken if a certain condition was known to exist 
(Cause-Correction, or C-C). 

The correctness of the responses of the pre-test subjects was not as 
important as the ease with which the subjects understood and responded to 
the questions . The experience gained in the pre-test was very helpful In 
determining which questions were not useful and should be eliminated. It 
was also helpful in determining the way in which the remaining questions 
should be streamlined to improve clarity. All of the Improvements suggested 
by the pre-test were made and the end result was the refined, open-form 
knowledge survey which was used in conjunction with the GnT simulation 
studies . 

The open form survey, contained in Appendix D, consisted of 58 questions 
which called for short, written answers. Thirty questions were of the 
straightforward type, 11 were of the C-S type, 9 were of the S-C type, and 
8 were of the C-C type. The open-form survey measured overall pilot know- 
ledge, as w’ell as knowledge in the six areas listed in Appendix D. Scorlc^ 
of the survey was performed with the aid of the answer key also provided In 
Appendix P . Partial credit was awarded for answers which came close to 
those given in the answer key. 
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For evaluation of the paper and pencil simulation tests, pilot knowledge 
about systems was needed. This was accomplished through the use of the 
closed-form knowledge survey given in Appendix E. The survey consisted 
of 20 multiple choice questions. Nine of the questions were of the strai^t- 
forward type, 6 were of the C-C type, and 5 were of the C-S type. All of the 
questions came from the open form survey and were selected on the basis of 
their ability to discriminate between good pilots and marginal pilots. Some of 
the incorrect responses offered by subjects in the open form survey were used 
as "dummy" alternatives in the closed, multiple-choice form. 

Under the multiple choice format all subjectivity in scoring was removed, 
and the time spent administering and scoring was greatly reduced. The answer 
key is given in Appendix E . Scores were provided for the three pilot knowledge 
subscore areas also given in Appendix E, as well as for overall pilot knowledge. 
Results of the closea form survey are discussed in Chapter II covering the 
paper and pencil simulations. 

B. General Results of the Closed- Form Knowledge Survey 

The closed form knowledge survey was administered to forty oilot-subjects, 
thirty of whom were also participants in the pilot decision-making workshop. 

The mean total score for the forty subjects was 12.4 with total scores ranging 
from five to seventeen. The maximum possible score was twenty. 

Statistical tests were performed to determine if any relationships existed 
between knowledge survey scores and pilot background variables. The Spearman 
rank coefficient was used as a measure of correlation throughout. A summary 
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of the correlation coefficients between total knowledge survey score and 
pilot background variables appears in Table IV- 1. 

Table IV- 1 

Total Knowledge Score Spearman Correlation Coefficients 


and Observed Level of Significance (In Parantheses) 


Total Flight Hours 

. 131 

(.42) 

IFR Hours 

.002 

(.99) 

Single Engine Hours 

.467 

(.002)* 

Rating 

.430 

(.006)* 


♦means significant correlations at p .05 level 


As seen in Table IV- 1, almost no correlation exists between total knowledge 
survey score and total hours or IFR hours. These lack of relationships suggest 
that accruing general flight experience or IFR flight experience does not 
guarantee knowledge will also increase. One possible explanation for this 
observation, however, is that, as pilots accrue more and more flight time, 
they tend to advance to more sophisticated aircraft with sharply different 
operational characteristics. The knowledge survey was aimed at the single 
pilot IFR operations common in light aircraft. These two relationships may 
not be as strong, then, because pilots with more flight experience may have 
moved out of the scope of the knowledge survey. 

Substantial positive correlations are seen, in Table IV- 1, between total 
knowledge survey score, and single engine hours and rating. These two 
relationships lend support to the knowledge survey's validity as a general 
tool to measure knowledge of single-pilot IFR operations and aircraft systems. 


22 




One would expect increased exposure to single engine operations (more single 
engine hours) would also increase a pilot's knowledge of single engine operations 
(which was the focal point of the knowledge survey). Additionally, as one's 
tested level of competence increased, knowledge should also increase. 

The knowledge survey was broken into three subcategories: Engine and 
Fuel Systems, Electrical Systems and Cockpit Instrumentation, and Weather 
and IFR operations. They were named Category I (CATSCR 1), Category n 
(CATSCR 2), and Category III (CATSCR 3), and contain 7, 7, and 6 items, 
respectively. The means and range of sco 'es for each category for the 
forty subjects are given in Table IV-2. 


Table IV-2 

Mean Scores and Range of 
Scores for Categories I, n, and HI 

mean % 
(%of max 
mean possible) 

Category I (maximum possible = 7) 4.82 68.9 

Category n (maximum possible = 7) 3.750 53.6 

Category ni (maximum possible = 6) 3.850 64.2 


range 
low hl^ 

1 7 

2 7 
0 6 


The same correlation tests were applied to these scores as were per- 
formed on the total knowledge survey score. A summary of the correlation 
coefficients between the three category scores and pilot background variables 
is given in Table IV- 3. 
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Table IV- 3 


Correlation Coefficients and Observed Level of Significance 
Category I, n, and m Scores, and 
Background Variables (Levels of Significance In Parantheses) 



Total Fli^t 
Hours 

IFR 

Hours 

SE 

Hours 

Rating 

Category I Score 

.197 (.22) 

.090 (.58) 

.474 (.002)* 

.273 (.09)* 

Category II Score 

-.115 (.48) 

.194 (.23) 

.184 (.26) 

. 195 (. 23) 

Category m Score 

.120 (.46) 

-.001 (.99) 

.376 (.017)* 

.376 (.017)* 


♦indicates significant relationships at p < . 10 level 


As evident in this table. Category I score (engine and fu^l systems) is 
positively correlated with single engine hours and ratings, whereas no 
significant correlation exists between Category I scores and total flight time 
or IFR flight time. These results may be due, again, to the fact that the 
knowledge survey was aimed toward single pilot IFR operations. There are 
sharp differences in powerplants between the sophisticated airplanes exper- 
ienced pilots are more likely to fly and the simpler, lighter crafts flown in 
single pilot operations. This is particularly true when one considers the fact 
that higher performance airplanes are often powered by turbojets or turbo- 
propellers . 

Category HI score (W'eather and IFR operations) is positively correlated, 
again, with single engine hours and rating, and uncorrelated with total flight 
time and IFR hours. No correlations were foimd in any ease involving Category n 
(Electrical Systems and Cockpit Instrumentation). 
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A siimmary of individual subject knowledge survey performance, including 
results ^or each of the three subscores is foiuid in the Master Data Table, 
Table VI-4 in Chapter VI. 
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V. GAT FULL MISSION SCENARIO (FMS) RUNS 


Prior to the development of the final paper and pencil experiments, a 
series of full mission simulation (FMS) experiments were performed. These 
experiments, which are described below, provided background for designing 
paper and pencil scenarios and a benchmark which such scenarios coulu be 
matched for a rudimentary cost/benefits evaluation. 

A. Purpose 

A Singer GAT-1 flig^Jt trainer was reconfigured to r 'rve as a fli^t simu- 
lator for use in "LOFT" t 3 rpe scenarios. These scenarios each involved a 
critical in-flight event imbedded within an otherwise normal simulated IFR 
flight mission. The purpose was to gain an understanding of: 

a) how pilots of different backgrounds assess and respond to 
such instances: 

b) the psychological stress of in-fli^t events, appropriate cq)ing 
processes, and the modeling of such processes; and 

c) the interaction (hat exists between air traffic controllers and pilots 
during CIFE's. 

In keeping with the full mission scenario approach, each subject went 
throu^ a pre-fli^t planning phase involving a complete weather briefing, 
route planning, and filing of fli^t plan. Take-off, climb and enroute phases 
of each scenario began under normal IFR operating conditions. Real time 
ATC communications, including background conversation were used to 
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enhance realism. A critical event was introduced some twenty to thirty 
minutes into each simulated flight. 

The conduct of an FMS is outlined in the paragraphs which follow. Com- 
plete operating instructions and detailed supporting material are contained 
in the Master Notebook for GAT Scenarios, a copy of which is available ta 
this project's file at NASA-.Ames. 

B. Experimental Equipment 

The primary piece of equipment used in the full mission simulation studies 
is a Singer General Aviation Trainer (G-AT-H on a motion base. Three degrees 
of freedom, roll, pitch, :md yaw, are provided by the motion base. This 
machine simulates, both in design and performance, a t^’plc&l single engine, 
carbureted, fixed pitch prop, fbced gear aircraft. The avionics equipment 
includes dual navigation and communication radios, dual V’OR indicators (one 
with glldeslope), an automatic direction finder, an audio control p.anel, and a 
three- lig^t marker beacon receiver. 

Modifications have been made to the standard GAT cockpit. A trans- 
ponder and a digital clock have been added to the instrument panel. .A fuel 
selector switch has been installed to the left of the pilot's scat. A lapel 
microphone has been added to pick up the pilot's communications and cockpit 
sounds. Two floodlights and a closed circuit television camera have been 
mounted over the pilot's right shoulder to view the Instrument panel. 

E.xternal modifications h.ive also been made to facilitate the experiment. 

The windows of the G.AT have been covered with a one-way reflective film 
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(Scotchtint) so Hiat the pilot can be observed daring the fli^t without his 
knowledge. A display for showing which fuel tank is active has been installeu, 
as well as external controls for the ammeter, panel li^t intensity, and for 
power (rpm) reduction. These are all new additions to the standard GAT 
hardware. 

The experimenter has the capability to control the operational status of 
some of the GAT systems, and to determine the values of key parameters. 

The following can be rendered inoperative: attitude gyro, directional gyro, 
altimeter, airspeed indicator, turn coordinator, vertical speed indicator, 
VOR/LOC indicators, automatic direction finder indicator, glides lope, 
and engine. Additionally, oil pressure, oil temperature, cylinder head 
temperature, fuel level for each tank, engine sound volume, gross wei^t, 
center of gravity, outside air temperature, rou^ air magnitude, barometric 
pressure, and wind direction and velocity are subject to continuous control. 
An X-Y plotter connected to the GAT tracks the progress of the fli^t on an 
enroute low altitude chart, and provides the air traffic controller with the 
equivalent of radar fli^t monitoring. 

Communication channels have been wired to permit t«’o-way communica- 
tion between the Jumpseat*and the ATC monitoring station. It allows pilot 
activities such as fre quency changes to be relayed to ATC by the Jumpseat 
observer as augmentation to video monitor viewing. It also provides for 
ATC cueing of Jumpseat for changes in environmental GAT parameters and 
introduction of systems failures. 

'■Jumpseat refers to an experimenter who rides outside the cockpit but who 
can both observe pilot cockpit behavior and also initiate system failures. 
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The equipment described above helps to provide fidelity and realism for 
the subject, adequate experimental control of the fli^t environment, and 
audio-visual recording of experimental fli^t data. 

C. FMS Procedures 

The following materials support a GAT scenario experimental session 
from initial contact of subject to raw data collection. T}q>ically, three 
experimenters are required to execute a session. One (Director) handles 
subject contact before and after die simulated fli^t. .Another acts as 
ATC during the run, and a third sits in the GAT Jumpseat to control 
cockpit conditions and to call out instrument status. The general pro- 
cedure for a GAT scenario experimental session follows: 

1) Subject contacted, explanation of study read, appointment is 
made, aircraft manual and subject background data form are 
mailed to Subject. 

2) Subject arrives and is met by Director. Sidiject is taken to a 
briefing room, where he initiates fli^t planning. 

3) Meanwhile, ATC prepares control station and Jumpseat pre- 
pares GAT with detailed checklists. 

4) Director prepares GAT room conditions and sees that all 
checklists are completed. 

5) When Subject finishes planning. Director escorts him to GAT 
room and familiarizes Subject with GAT cockpit. 

6) Director has Subject start the engine and closes cockpit door. 
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Jumpseat controls engine status gauges, winds aloft, and 
various other environmental conditions as cued by ATC or 
Director. Jumpseat also monitors and reads instruments that 
are difficult to read from the video camera. This aids in later 
review of the video ta^ and also aids ATC in determining which 
comm frequency has been selected. 

Director fills out data form for the particular run including 
the clock times for significant events to aid subsequent video 
tape reviews. He also obtains Subject performance judgments 
from ATC and Jumpseat at several points in the scenario. 

10) After Subject lands the aircraft. Director meets Subject in cock- 
pit and takes Subject to debriefing room. 

11) Subject discusses the flight with Director, answering specific 
questions concerning the CIFE. The debriefing is recorded on 
audio tape. 

12) Meanwhile, ATC and Jumpseat shutdown GAT and supporting 
hardware, and document and store raw data. 

D. FMS Scenarios 

Three separate full mission scenarios have been created. Each scenario 
has accompanying support material in terms of charts, e.xperlmenter check- 
lists, ATC scripts, pre-recorded background communication tapes smd data 
forms. Samples of these support materials are contained in the Master Notebook. 
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Each of the three scenarios features a different t 3 rpe of critical in-flight 
event. Scenario 1 involves a loss of fuel from one tank. Full power is 
recoverable by switching tanks at which time the pilot must decide on one of 
several destination alternatives. Scenario 2 involves a partial power failure. 
No actions are available to restore full power to the aircraft. The pilot must 
decide on one of several destination alternatives or an emergency landing. 
Scenario 3 involves a partial navigation system failure during an ILS 
approach. The pilot must recognize the failure and select an appropriate 
alternate approach procedure and/or airport. All scenarios feature weafljer 
near IFR minimums and a mix of mountainous, flat, and seacoast terrain. 
Details of each scenario appear below. 

Scenario 1 

The objective of this scenario is to reveal how a pilot responds to inadver- 
tent loss of fuel in flight, resulting from the over-wing siphoning of fuel through 
an improperly sealed filler opening. Of particular interest are, 1) his actions 
to restore engine power when the fuel supply from the tank in use is depleted, 

2) his decision on where to land in view of the unanticipated reduction of 
remaining fuel and 3) his aircraft control performance prior to and aft the 
CIFE. 

Each subject is instructed to prepare and file an IFR flight plan for a night 
fli^t from Seaport Beach to Mountaindale airport. The weather at the point 
of departure and along the route of fli^t is IMC (ceilings are less than 1000 
feet and visibilities are less than three miles). At the destination airport 
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the weather is marginal VMC. Seaport Beach is on the coast: Mountain- 
dale is surrounded by mountainous terrain. 

After takeoff the flight proceeds along a predetermined route as specified 
in the original clearance (radar vectors to the Seaport 259° radial to Ranch 
intersection, Victor 97 to Goadiill VOR, direct). As the flig^it continues 
along this route, the pilot is instructed to contact the J^propriate controlling 
facilities. The fuel supply in the tank in use is reduced gradually, but at a 
rate much faster than that of normal consumption. When the flight reaches 
a certain point, the fuel supply in die tank being used is depleted, and the 
engine sputters and dies. At the time of the engine failure, the flight is in 
instrument conditions, experiencing moderate turbulence, and not in radar 
contact. 

The only action the pilot can take to regain engine power is to switch 
fuel taidcs. In the course of solving thi;. p' ‘‘blem, the pilot must set 
priorities concerning the activities he deems appropriate. Once the 
pilot switches tanks, and engine power is restored, normal operations 
can be continued. However, the flight now has half the original hiel. In 
view of this new limitation, the pilot must decide on whether to continue or 
to divert to an alternate. There are three alternatives from which the pilot 
must choose: he can continue on to his destination, land at a closer air- 
port, or return to the point of departure. The fli^t has fuel sufficient 
to fly to and land at any of the alternatives, but his choice is complicated 
by varying weather conditions at the different airports, the different distances 
and times to fly to the airports, and the pilot's perception of the problem. 
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Scenario 2 


lliis scenario simulates the reduction of available engine power due to 
a broken baffle in the muffler during a cross-country IFR flight. 

The mission is to fly from Seaport Beach to Moimtaindale. Immediately 
prior to takeoff, the pilot is cleared along a route approximately parallel 
to the one which he had filed. At the time of departure the Seaport Beach 
weather is IMC (ceiling is 1100 feet and visibility is two miles in rain, fog, 
drizzle) and the Mountaindale weather is marginal VMC. 

As the flight progresses, moderate turbulence is encountered near 
Singer intersection with a taUwind at thirty knots. When the fli^t pro- 
ceeds past Thermal intersection, engine power is linearly decreased to 
1500 rpm over a period of three minutes . This is accompanied by tachometer 
inc ications and a decrease in engine sound. Simultaneously cylinder head and 
oil temperature are increased to maximum level. The power level is not 
sufficient to maintain the enroute altitude, so a descent begins as the power 
loss continues. The problem consists of inadequate power and rising terrain 
while out of radar contact in instrument conditions. 

At this point the available alternatives are: 1) continue to Mountaindale, 

2) return to Seaqwrt Beach, 3) land on the immediate terrain, 4) land at 
Singer, 5) land at Wind Falls, 6) land at Link County, and 7) land at Pelton 
Naval Air Station. A major decision is whether or not to declare an emer- 
gency, especially since the assigned altitude cannot be mail ‘^ained. T 3 i)ically 
the subjects proceed to Mountauidale or return to Seaport Beach. 
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Scenario 3 


The purposes of this scenario are 1) to reveal how a pilot, without the 
aid of warning lights or flags, determines that an essential part of his 
approach navigation equipment (localizer) has failed, and 2) to reveal what 
decisions and actions he makes to complete the fli^t in view of the aircraft's 
new status. The pilot is instructed to depart Mountaindale airport, to con- 
duct two ILS approaches at Mountaindale, and to land at Mountaindale after 
the second approach. His fli^t plan specifies the route of flight to be "via 
radar vectors". The local weather conditions during departure and the two 
subsequent instrument approaches are "ceiling 500 feet overcast, visibility 
two miles in rain and fog, wind from the east at ten knots. " 

After takeoff the pilot is vectored along a predetermined route to inter- 
cept the localizer course for the runway five ILS approximately five miles 
from the outer marker. After completion of the fi^st approach, the pilot 
is vectored around to intercept the localizer for his second, and final 
approach. As the localizer needle sweeps to the center position during 
initial interception, it is rendered inoperative. (In this mode the localizer 
atedle remains idle in the center position with a ’TO" indication. ) At the 
time failure, the flight is in instrument conditions, in radar contact, and 
experiencing light turbulence. 

The pilot can use ATC position information, ADF crosschecks, or note 
that C.e needle is stationary to determine the localizer needle has failed. 

Upon confirming its failure, the pilot then must decide what to do next. Ke 
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could conduct an NDB approach, a VOR aH>roach, or divert to another air- 
port. Of these alternatives, the VOR approach is the only feasible one. 


E. Subjects 

Twelve sid>jects were selected for the FMS experiments. Four were used 
in each of the three scenarios. Their ages ranged from 21 to 56 years old. 
Although all of the subjects were instrument rated, their licenses covered 
the spectrum from Private to ATP. Six of the twelve held CFI ratings and 
five held turbine ratings. In terms of their primary flying activities they were 
equally divided (six each) into pleasure and professional flying groups. Their 
total flying hours logged ranged from 270 to 8800 hours. Table V-1 summarizes 
these data. 


Table V-1 

FMS Subject Background 


Scenario 

Subject 

Age 

Licenses 

Total Hours 

Type of Flying 

1 

1 

36 

Pvt. 

420 

Pleasure 


2 

56 

Comm/ 

CFI 

5000 

Pleasure 



42 

Comm 

1200 

Business 


4 

46 

ATP 

8800 

Business 

2 

5 

23 

Comm/ 

CFI 

1550 

Professional 


6 

34 

Comm 

5000 

Pleasure 


7 

34 

Comm/ 

CFI 

3000 

Pleasure/Mil. 


8 

30 

Pvt. 

300 

Pleasure 

3 

9 

31 

Comm/ 

CFI 

1750 

Professional 


10 

22 

Pvt. 

270 

Pleasure 


11 

21 

Comm/ 

CFI 

480 

Professional 


12 

21 

Comm/ 

CFI 

600 

Professional 
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As noted in Table V-1 an attempt was made to obtain a mix of exper- 
ience and ratings for each of the test scenarios. All subjects were unpaid 
volunteers from the Columbus, CXiio area. 

F. Data Collection 

As noted in Figure V 1, three major types of performance data were 
collected for each FMS run. 

1) 'Stick and rudder" performance, i.e. , basic control of heading, 
altitude, and airspeed 

2) Communications 

3) Response to the CIFE 

Stick and rudder performance was evaluated both objectively and sub- 
jectively. Subjective ratings on a scale of one to seven were given for navi- 
gation skills and attitude control by each of the three experimenters present 
during a run. (All e^erimenters were qualified pilots as well as researchers. ) 
Subjective rating averages for both navigation and attitude control skills ranged 
from a low of 1.2 to a hi^ of 6.7. There appeared to be a high correlation 
between the two ratings, i.e. , a subject with good navigation skills also 
exhibited good attitude control skills as noted in Figure V-2. Only ten subjects 
were rated due to unscheduled equipment malfimctlon during a portion of two 
runs. 

A more objective indication of stick and rudder performance was obtained 
from time traces of altitude, airspeed, and heading deviations covering the 
period immediately surro'mdlng the introAiction of the CIFE. These data 
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Figure V- 




were obtained by analyzing video tapes and coordinated audio tracks for 
each FMS run. Samples of each data are contained in Figures V-3 and V-4. 
Plots for all subjects are contained in Appendix F. 

Communication skills were also evaluated both subjectively and 
objectively. Each observer rated each subject on a scale of one to seven. 
Average scores here raided from a low of 2.0 to a high of 6.3. In addition 
to those ratings, which were made at the time of the experiment, complete 
transcripts of communications were prepared after the fact from the audio 
tapes . A portion of one such transcript for the second scenario around the 
time of the CIFE has been reproduced in Figure V- 3. These transcripts per- 
mitted a detailed analysis of interactions between pilot and controller as well 
as an indication of the information search by subjects. 

The third indication of performance was the actual decision making 
response of subjects when faced with a CIFE . A standard data sheet was 
used to summarize the observed behavior of each subject. Problem detection 
and diagnosis as well as decisions and actions were noted (see Figure V-6). 
The information used to complete these sheets was obtained by studying 
the video tapes, consulting observers* data sheets and from a tlorough 
(tape recorded) debriefing of each subject after his FMS run. Observed 
stress was a subjective estimate (scale one to ten) by the experimenters. 

Pilot criticality estimates were made by the subjects (scale one to ten) and 
were intended to indicate the degree of criticality each placed on the CIFE . 
Flying time estimates were made by the subjects who were asked how long 


38 



NAVIGATION SKILLS 


Figure V-L. Skill Ratiags For FMS Subjects 



39 



Figure V-3. FMS Poor Stick and Rudder Performance 





Figure V-4. FMS Good Stick and Rudder Performance 




Figure V-5. Communications Transcript, Subject 7, Scenario 2 


15.59 

15.71 
16.28 

16.40 

16.71 

16.81 

16.96 

17.06 

17.17 

19.20 

19.33 

19.45 

19.52 

19.74 

19.84 

20.48 

20.60 
20.66 
20.73 
20.81 

21.96 

22.07 
22.80 

22.98 

23.05 

23-22 

23-31 

23.51 

23.59 

23.65 


23.85 

24-07 

24.31 

24.40 

24.82 


25.10 


C: Japan Air 231, contact East Bay Center, 132.15 heading. 

B: Japan Air 231 to 132.15, good day. 

C: Uh, one niner two golf papa, roger, maintain 6000. 
(unintelligible) . 

P: One niner two golf papa, roger, maintain 6000. 

P: Center, November one niner golf papa reporting Thermal 
at this time. 

C: Two golf pop's at Thermal intersection. Thanh -you, sir. 

B: East Bay Center, United 694 climbing to one-two-i_ro. 

C: United 694, radar contact, climb unrestricted to flight 
level three-seven -aero. 

B: Unrestricted to three-seven-zero. United 694. 

C; Uh, one niner two golf papa, radar contact, uh, 
(unintelligible) two miles north of Thennal intersection. 

P: Uh, roger, one niner two golf papa. 

C: Answer golf papa, you can expect, uh, 8000 in ten miles. 

P: Two golf papa, roger. 

C: King Aire 90 Fox Hotel, contact Seaport Approach 119-6 
B: 119.6 for 90 Fox Hotel. 

B: East Bay Center. Centurion 5343 Foxtrot climbing to .7000. 
C: 43 Foxtrot, East Bay Center, ident. 

B: 43 Foxtrot, roger. 

C: 43 Foxtrot, radar contact, proceed on course. 

B: 43 Foxtrot, on course, roger. 

C: Pacer 62, contact Pel ton approach 126.2. 

B: Pacer 62 to 126.2. good day. 

B: Center, this is Baron 3622 Tango. Has anyone reported 
turbulence on vector two -twenty -two to the northwest here? 

C: 22 Tango, that's negative, sir.- 

B: We're in moderate turbulence at 12,000, and picleing up 
mixed ice. Any chance of one four thousand for 22 Tango? 

C: Baron 3622 Tango, climb and maintain one four t'uousand. 

B: 22 Tango leaving 12.000 foj^ one four thousand. 

P: Center, November one nine two golf papa. 

C: One niner two golf papa, go ahead, sir. 

P: Okay, roger sir, experiencing ,uh, difficulties with 
my engine. I'm losing RPM and request, uh, immediate descent 
to the nearest airport. 

C: One niner two golf pop, uh, stand by. 

C: Two golf pop, uh, all the airports in the vicinity are 
IFR. You're currently, uh, five miles northwest of Thermal. 
P: Uh, roger, what's, uh, what^'s that weather at LinX? 

C: Okay, uh, stand by. I'll have it for you in just a second. 
C: Yeah, two .golf pop, uh. Link weather at zero 300. Uh, 

500 scatte-^ed measured 800 overcast two miles rain and fog. 
Uh, wind one six zero at ten, altimeter two niner point 
four five. 

P: Uh. roger, and Center be advised, uh, one niner two 
golf papa is, uh, losing altitude at this time, unable to 
maintain altitude, and RPM is dropping off. Uh, request 
vectors for the clearest weather possible you can find. 

I'm gonna have to be setting it down. 

C: Uh, two golf papa, understand (unintelligible). Unable to 
maintain altitude, requesting vectors. All airports in the 
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Figure V-6. Scenario 1 - CIFE Response 
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Figure V-7 . Decision Factors Rating 
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they diou^t they could continue to fly after the CIFE. The rest of the data 
sheet entries were filled by experimental observation or subject statement. 

In an attempt to probe their personal rationale, each s\ibject was also 
asked to complete a rating form covering some 21 separate factors which 
may have influenced his decision in the face of the CIFE. Each factor was 
rated by the subject on a scale of one to seven. (Ratings for Scenario 1 are 
shown in Figure V-7. ) 

The final piece of information collected was the test score from an open- 
form knowledge survey. This survey was used as a pilot study to help develop 
the closed-form knowledge survey used with the paper and pencil scenarios. 

All subjects for scenarios one and two also participated in the paper and 
pencil tests. They were identified in the master data sheet with "1" in the 
GAT column. Complete data summary sheets for all three scenarios are 
contained in Appendix F. 

G. Performance Evaluation 

Because of the small sample size and differences across scenarios, 
it was difficult to develop solid statistical information concerning pilot per- 
formance in such full mission simulation studies. However, by analyzing 
the data mentioned above, it became apparent that the subjects in these 
experiments possessed n wide range of cockpit management styles and skill 
levels. Although difficult to quantify, "good performance" was easily recog- 
nized by both on-site observers of the FMS runs and others who examined 
the various data collected from those runs. The elements of "good performance" 
included: 
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1) professional use of the radio 

2) precise heading and altitude control 

3) constant awareness of the aircraft position along its 
intended route 

4) prompt, but not necessarily instant, response tc he onset of the 
CIFE (detection) 

3) systematic procedure for trouble-shooting 

6) diversion decisions which allowed for further imcertainties 
Evidence supporting each of diese six characteristics of good performance 
can be found in Figures V-2 to V-7 above. For example, consider Figures V-3 
and V-4 which depict what appear to be good and poor stick and rudder perform- 
ances. The time traces for subject 4 exhibit very small unplanned deviations 
in airspeed, altitude and heading both before and after the onset of the CIFE 
(loss of fuel cap). Subject 3, on the other hand, demonstrates a somewhat 
unstable control of these three fli^t parameters even before the onset of the 
CIFE. Furthermore, during and after the CIFE, his airspeed, altitude and 
heading excursions appear to iiK:rease in both frequency and amplitude which 
may indicate that he was loaded beyond his ability to cope with the problem at 
hand. Coincidentally, it is also easy to find evidence that subject 4's per- 
formance in each of the six elements listed above was generally superior to 
th It of subject 3. Furthermore, there is supporting evidence that the "good 
performers" tend to score higher on both forms of the knowledge survey than 
do the "poor performers". 
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Since much of the evidence of FMS performance is anecdotal, a brief 
narrative description of each subject's actions and their characteristics 
has been prepared. T>ese narratives are contained in Appendix F. These 
narratives include comments on each subject's background, personal char- 
acteristics, and management style. They are perhaps the richest information 
source for gaining insists into how these twelve sid>jects made use of 
available resources in the face of critical in-fli^t events. 

H. FMS Conclusions 

The sample was too small to provide any&ing other than some initial 
hypotheses concernii^ pilot performance in such a full-mission setting. 
However, the following tendencies were noted: 

1) Cockpit management style varies widely among pilots. For 
example, some are extremely self-reliant, others want 
immediate and extensive help from ATC while still others 
make the decision making process a joint effort with ATC. 

2) Good stick and rudder pilots seem to have excess capability 
and maintain good stick and rudder performance during and 
after the CIFE. More marginal stick and rudder pilots, on 
the other hand, show increased frequency and amplitude of 
heading and altitude excursions, and experience communication 
difficulties in the face of a CIFE . 
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3) Pilots who score well on the knowledge test instruments 
tend to perform well in problem diagnosis and decision 
making. (GAT subject performance on the paper and pencil 
tests are discussed in Section VI-L. ) 

From the observations of the experimenters and comments made by 
participating subjects, it appears that such a full mission simulation exer- 
cise, coupled with an appropriate knowledge survey and debriefing, could 
be a valuable tool for recurrent training of IFR pilots. 
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VI. PAPER AND PENCIL SCENARIO TESTS 


The GAT FMS scenarios were extremely valuable In gaining a better 
understanding of how pilots make decisions in the face of CIFE's. However, 
they were very expensive to run, in terms of equipment, subject and exper- 
imenter time, and the data was difficult to analyze in objective fashion. The 
paper and pencil scenario concept was developed to provide a more econom- 
ical way to study the CIFE phenomenon and to reduce the data collection and 
analysis problems inherent with FMS experiments. 

A. Background 

The paper and pencil (P/P) concept was tested in two different ways prior 
to full-scale implementation. First, two pilots, bodi on the aviation faculty 
at The CXiio State University and considered to be experts in their field, 
evaluated several GAT subjects' decisions on two of the three GAT scenarios. 
The two experts then made their own diagnoses and decisions on the third. 

From these sessions it became clear that pilots could diagnose problems and 
make diversion decisions in a P/P format. Furthermore, the expert pilots 
found the tasks more realistic when they injected themselves into the 
scenario, rather than playing the role of observer. 

A second P/P format pre-test was run with a local aircraft mechanic who 
is widely respected as an expert. The purpose of this exercise was to determine 
if someone could diagnose a mechanical failure in an interview situation. The 
mechanic was given the initial symptoms to the problem and was asked to arrive 
at an explanation of the cause. He asked questions about the status of various 
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indicators and hypothesized aloud as he systematically eliminated potential 
causes . The interviewer provided readings from instruments and answers 
to sundry status inquiries verbally. The mechanic had no trouble diagnosing 
the problem in the interview format. These results suggested many of the 
techniques used in the full-scale study. A transcript of part of that Interview 
is contained in Appendix K. 

In order to facilitate analysis and to eliminate interactions, it was decided 
to break the paper and pencil testing into two distinct elements; one set of 
scenarios directed toward problem diagnosis and a second set directed toward 
pilot decision making based upon a common diagnosis of the problem. 

The diagnosis scenarios were conceived to meet several important criteria: 

a) a system or component failure that would be nondeteriorating over time, 

b) insoluble (at least while in the air), but identifiable, c) precipitated by com- 
ponent failure or weather conditions, and d) Important enough to require a 
subsequent diversion decision. There also had to be enough evidence within the 
available Information to unambiguously identify the cause of the problem. 

Once the four problems were selected for use, the concomitant S 3 rmptoms 
and instrument readings were verified with the expert airplane mechanic 
referred to earlier. The given symptoms for the problems were selected to 
lead the subject in the general correct diagnostic direction, but were insufficient 
for trivial solution. The four scenarios selected Involved: 1) an oil leak at 
the oil pressure gauge line, 2) a vacuum pump failure, 3) a magneto drive 
gear failure, and 4) a frozen static port. The diversion scenarios designed to 
illuminate a pilots decision making strategies are discussed In Section E below. 
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B. The Testing Procedure 


The procedure used in the paper and pencil scenario (PPS) testing required 
about ninety minutes. The period was used for four major data collection inputs 

a) Biographical Data (See Appendix H) 

b) Closed-Form Knowledge Test (See Chapter IV) 

c) Diagnostic Performance on Four Different Scenarios 

d) A Destination-Decision Problem Dealing With Information Seeking 
Strategies 

These will be discussed in detail in the following sections. 

Announcements were posted at local flying clubs and fixed base operators 
(FBOs) to attract volunteer subjects from the flying community. Interested 
IFR rated pilots called in for details and were scheduled for one of several 
two-hour sessions. In addition, qualified pilots from The Ohio State University 
and local communities were called by telephone and invited to participate. 

Each session proceeded as follows: Participants gathered in a large con- 
ference room. After a brief introduction by one of the principal investigators, 
subjects filled out the biographical forms and took a closed-form knowledge 
survey. A briefing statement covering scenario weather, airspace and the 
airplane to be ’^lown" was given the subjects while they looked at enroute 
charts and weather maps (see '.opendix H). The subjects then went individually 
to separate rooms with an experimenter. Here, they went through the problem 
diagnosis and diversion-decision excerclses for about one hour. 

The instructions were read to the subjects (see Appendix H) which explained 
how the four problem diagnosis scenarios would be run. For each problem 
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diagnosis scenario, a brief mission introduction was read, identifying area 
weather, fli^t origin and destination, and referring to a low altitude enroute 
chart with the airports highlighted. Following the introduction, symptoms 
for the problem were given (e.g. "After twenty minutes of routine flying you 
notice the smell of hot engine oil"). At this point the subject was signalled 
to begin his diagnosis by the question, "What would you do?". A stop-watch 
was started when the subject began his information search, allowing four 
minutes for completion. 

While referring to a modified diagram of the Piper Arrow instrument 
panel, subjects began to ask the experimenter for pieces of information which 
could be collected by the pilot if he were actually in the cockpit of a Piper 
Arrow. In addition to readings from flight instruments, engine gauges and 
navigation/communication radios, the subject could query the experimenter 
for information concerning structural ice formation, noise, cabin conditions, 
status of the cabin interior, and system response to control settings 
for throttle, mixture, RPM, fuel selector, etc. When queried, the experi- 
menter looked up the information on tvio sheets of paper which followed a 
standard format. After finding the requested information and telling the 
subject, the experimenter then noted the item with a number on the sheet. 

The numbers denoted the sequence of queries such that the order cc -Id be 
reconstructed. A third sheet was available for noting hsrpotheses of potential 
causes mentioned by the subject during the information search. Their position 
in the sequence was also noted. 
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The clock was stopped when the subject indicated that he had discovered the 
cause of the problem. If the four minutes ran out, the suoject was asked to 
make a best guess as to the problem's source. The time taken was recorded 
and the subjecc was asked how long he thou^t the plane would fly with such a 
problem. He was then ashed to judge the criticality of the problem as he had 
diagnosed it, on a scale of one to seven. An explanation of the cause of the 
problem was chen read to the subject and the final two estimates were repeated. 
This procedure was repeated for four different scenf‘rios and took aboui 
25 minutes to complete. 

Fortj' volunteer subjects participated in the p/P scenario study. All 
but one were instrument-rated and with experience ranging from 160 to 
19,400 total flying hours. Nineteen had commercial licenses and twelve had 
Air Transport ratings. Eight of the subjects had participated earlier in the 
GAT-1 study- Subject background data is show.: in Table VI-1. Figures 
VI- 1 to VI-3 depict the fli^t experience of the subjects. Table 71-2 summarizes 
subject data fot background data by frequency and percent. Figure VI-4 depicts 
the subjects scores on the closed-form of the knowledge test. It Is wort’' 
noting that the scores were surprisingly low considerii^ the fact that the 
mean number of hours experience was 3823 hours. 

C. Pilot Background Data and Diagnostic Data Collection 

Pilot background data were coded into seven variables. The fc.ir continuous 
numeric variables were: score on the knowledge survey (0-20), tXat ilylrg 
hours, total single-engine hours, and total IFR (including actual, 
lated and time flown under IFR) hours. T'i. Tscrete variables were: 
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MIDPOINT TOTAL HOUIIS 
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PEimenTAOE BAR CHART 
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NIDPOIMT SCOitK OHf KNOWLEDGE TEST 



Table VI-2 


Freqaency Anal 3 rsls of Pilot Background 

Frequency 

Rating: 


Private 9 

Commercial 19 

Air Transport 12 

Training: 

Military 10 

CiviUan 30 

Most Frequent Flying: 

Airline 2 

GA Commercial 12 

Business 11 

Military 6 

Pleasure 9 


Percent 


22.5 

47.5 
30.0 


25.0 

75.0 


5.0 

30.0 

27.5 

15.0 

22.5 
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rating (private, commercial, and ATP), primary fli^t training (military or 
civilian) and the most frequent type of flying (airline, GA comm, business, 
military, and pleasure). 

Diagnosis scenario performance was coded into ei^t numeric variables 
for each subject on each scenario. These were: 

I - number of inquiries or control actions 

TT - total tracks (lines of coherent questioning) 

UT - unique tracks (tracks not repeated) 

C - correctness of final diagnosis (0-5) 

Z - correctness/total tracks 

E - efficiency = 25-2 z (minutes required) - 1-2 

CORINQ - correctness /total inquiries 

M - merits(C x E ) 

CB - criticality estimate before solution given 

CA - criticality estimate after solution given 

CNTRL - number of control actions taken 

The totals for these eight variables, summed across the four scenarios were 
also calculated and named as variables: 

TOTINQ - II + 12 + 13 + 14 
TOTTRAKS - TTl + TT2 ^ TT3 + TT4 
TOTUTRKS - UTl + UTL ^ UT3 + UT4 
TOTCOR - Cl ^ C2 " C3 + C4 
ZT - TOTCOR/TOTTRAKS 

TOTEFF - El * E2 + E3 ^ E4 
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CORINQT 


TOTMERTT 

TCRITBEF 

TCRITAFT 


TOTCOR/TOTINQ 
M1 + M2 + M3 + M4 
CBl + CB2 + CBS + CB4 
CAl + CA2 + CAS + CA4 


See the Glossary, Table VI-S. The diagnostic data, knowledge scores, 
pilot background data and decision data (see Section E below) were compiled 
and used in the total analysis. 


D. Diagnostic Performance 

Means and standard deviations for all performance variables are listed 
in Table VI-4. Comprehensive scores of total correct and total merit are 
shown as percentage distributions in Figures VI-5 and VI-6. The total correct 
distribution appears somewhat negatively skewed, while that for total merit 
appears to be normal. 

Groiqi performance on the four scenarios improved in terms of correctness 
and merit with the order of presentation, although all four problems were 
judged to be equally difficult to diagnose. This fact demonstrates some 
learning and strategy development by the subjects . 

When the pilot sample is broken down by rating, several differences emerge 
on various diagnc'ii' ; jrformance dimensions (see Table VI-5). Total correct 
and total merit scores increase as the level of certification goes up (Pvt. , 
Comm., ATP) consistent with "unventional wisdom. Performance on scenario 1, 
(the oil leak) seems to nm counter to presumed knowledge by the higher certifi- 
cate holders. However, scenario 2 (vacuum pump failure) and S (magneto 
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Table Vl-3 
Glossary 


1* AGE: Age of the subject - categorized into intervals: 

1) age 30 yrs. 

2) 30 yrs. <• age < 50 yrs. 

3) age > 50 yrs. 

2. AIRPORTS: Airports the pilot was willing to pass to locate proper repair 

facilities. 

3. AP: Variable for airports used in computer runs valued (0) if airports < 2 

and (1) if airports > 2 . 

4. APP: Approach attribute of an airport. Includes ILS vs. NOB approach. 

5. ATC: Air Traffic Control attribute of an airport presence of radar). 
^App’ Pilots importance assessment of approach attribute of an airport. 

7. Pilots importance assessment of an air traffic control attribute of 
an airport. 

8. Pilots importance assessment of time. 

9. B^-^: Pilots importance assessment of weather. 

10. Cl: Correctness score on Scenario #1 (possible correct: 0-5). 

11. C2: Correctness score on Scenario #2 (possible correct: 0-5). 

12. C3: Correctness score on Scenario #3 (possible correct: 0-5). 

13. C4: Correctness score on Scenario #4 (possible correct: 0-5). 

14. CAl: Subjective criticality estimate of event in Scenario #1 after beii% pro 

vided with the answer (scale 1-7; l=lowest criticality). 

15. CA2: Subjective criticality estimate of event in Scenario #2 after being pro 

vided with the answer (scale 1-7; l=lowest criticality). 

16. CA3: Subjective criticality estimate of event in Scenario #3 after being pro 

vided with the answer (scale 1-7; l=lowest criticality). 
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17. CA4: Subjective criticality estimate of event in Scenario #4 after being 

provided with the answer (scale 1-7; l=lowest criticality). 

18. CATSCRl: First category score on knowledge survey - knowledge sub- 

score for engine and fuel systems (possible correct: 0-7). 

19. CATSCR2: Second category score on knowledge survey - knowledge sub- 

score for electrical systems and cockpit instrumentation 
(possible correct: 0-7). 

20. CATSCR3: Third category score on knowledge survey - knowledge sub- 

score for weather and IFR operations (possible correct: 0-6). 

21. CBl: Subjective criticality estimate of event in Scenario #1 before ieing 

provided with the answer (scale 1-7; l=lowest criticality). 

22. CB2: Subjective criticality estimate of event in Scenario #2 before being 

provided with the answer (scale i-7; l=lowest criticality). 

23. CB3: Subjective criticality estimate of event in Scenario #3 before being 

provided with the answer (scale 1-7; l=lowest criticality). 

24. CB4: Subjective criticality estimate of event in Scenario #4 before being 

provided with the answer (scale 1-7; 1=1 west criticality). 

25. CNTRLl: Number of inquiries which involved control movements in 

Scenario #1. 

26. CNTRL2: Number of inquiries which involved control movements in 

Scenario #2. 

27. CNTRL3: Number of Inquiries which involved control movements in 

Scenario #3. 

28. CNTRL4: Number of inquiries which involved control movements in 

Scenario #4. 

29. CNTRLTOT: Total number of inquiries for all four scenarios which 

involved control movements 

CNTRLTOT = CNTRLl + CNTRL2 + CNTRL3 + CNTRL4 

30. CORINQl: Ratio of correctness to inquiries for Scenario #1: 

CORINQl = Cl/ll 

31. CORINQ2: Ratio of correctness to inquiries for Scenario #2: 

CORINQ2 = C2/I2 
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32. C0BINQ3: Ratio of correctness to inquiries for Scenario #3: 

CORINQ3 = C3/I3 

33. CORINQ4: Ratio of correctness to inquiries for Scenario #4: 

CORINQ4 = C4/I4 

34. CORINQT: Ratio of total correct to total Inquiries for all four scenarios: 

CORINQT = (Cl + C2 + C3 + C4)/ai + 12 + 13 * 14) 

35. DELTACl: Change in subjective criticality estimate of event for Scenario #1 

after being provided with the answer; DELTACl = CAl - CBl 

36. DELTAC2: Chaise in subjective criticality estimate of event for Scenario #2 

after being provided with the answer; DELTAC2 = CA2 - CB2 

37. DELTAC3: Change in subjective criticality estimate of event for Scenario #3 

after being provided with the answer; DELTAC3 = CA3 - CB3 

38. DELTAC4: Change in subjective criticality estimate of event for Scenario #4 

after being provided with the answer; DELTAC4 = CA4 - CB4 

39. DIFl: Difference between number of total tracks and number of unique 

tracks in Scenario #1: DIFl = TTl - UTl 

40. DIF2: Difference between number of total tracks and number of unique 

tracks in Scenario #2: DIF 2 = TT2 - UT2 

41. DIF3: Difference between number of total tracks and number of unique 

tracks in Scenario #3; DIF3 = TT3 - UTS 

42. BIF4: Difference between number of total tracks and number of unique 

tracks in Scenario #4: DIF4 = TT4 - UT4 

43. DIFT; Difference between number of total tracks and number of unique 

tracks in all four scenarios: DIFT = TOTTRAKS - TOTUTRKS 

44. El; Efficiency score on Scenario #1: El = C25 - 2 (minutes to diagnose) - (11 -2)] 

45. E2: Efficiency score on Scenario #2: E2 = [25 - 2 (minutes to diagnose) - (12 -2)1 

46. E3: Efficiency score on Scenario #3: E3 = [25 - 2 (minutes to diagnose) - (13 -2)1 

47. E4: Efficiency score on Scenario #4: E4 = f25 - 2 (minutes to diagnose) - (14 -2)1 

48. FLY: Computer variable for the variable flying; takes values; 

(0) if flying = 1,2,3, or 4 = non-pleasure 

(1) if flying = 5 = pleasure 
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49 . 


50. 


51. 


52. 


53. 


54. 


55. 


56. 


57. 


58. 


59. 


60. 


61. 


62. 


63 . 


FLYING: Most frequent kind of flying. 
Valued: (1) Airline 

(2) Commercial 

(3) Business 

(4) Military 

(5) Pleasure 


GAT: Participation in general aviation simulation; 0 = did not participate, 

1 = did participate 

GATKl: Open ended knowledge test on GAT subjects - subscore on engine 
operations (possible correct: 0-7). 

GATK2: Open ended knowledge test on GAT subjects - subscore on fuel 
systems (possible correct: 0-7). 

GATK3: Open ended knowledge test on GAT subjects - subscore on electrical 
systems (possible correct: 0-7). 

GATK4: Open ended knowledge test on GAT subjects - subscore on cockpit 
instrumentation (possible correct: 0-7). 

GATK5: Open ended knowledge test on GAT subjects - subscore on weather 
(possible correct: 0-7). 

GATK6: Open ended knowledge test on GAT subjects - subscore on IFR 
procedure (possible correct: 0-7). 

GATKT: Average of all parts of open ended knowledge GAT test: 

GATKT = GATKl + GATK2 + GATK3 + GATK4 GATK5 + GATK6 

6 


GONOGO: Designates whether the pilot would have taken the flight under the 
given conditions. Valued: (0) - would not go, (1) - would go. 

II: Number of inquiries in Scenario #1. 

12: Number of inquiries in Scenario #2. 

13: Number of inquiries in Scenario #3. 

14: Number of inquiries in Scenario #4. 

IFR: Variable designating upper and lower quartiles of IFR hours: 

(0) if IFRhrs. :£ 175 

(1) if IFRhrs. 2700. 
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64. IFRHR5: Hours of flying under instrument flight rules. 

65. INPTRl: Ratio of inquiries to total tracks in Scenario #1: INPTRl = Il/TTl. 

66. INPTR2: Ratio of inquiries to total tracks in Scenario #2: INPTR2 = I2/TT2. 

67. INPTR3: Ratio of inquiries to total tracks in Scenario #3: INPTR3 = I3/TT3. 

68. INPTR4: Ratio of inquiries to total tracks in Scenario #4: INPTR4 = I4/TT4. 

69. INPTRT: Ratio of total inquiries to total tracks for all four scenarios; 

INPTRT = TOTINQ/TOTTRAKS 

70. KNOW: Variable designating upper and lower quartiles of KNOWLEDG scores; 

(0) if KNOWLEDG ^ 9 

(1) if KNOWLEDG 2: 16 

71. KNOWLEDG: Score on aircraft systems survey (possible correct: 0-20). 

72. LATELY: Relative amount of flying done in last year: 

(0) if pilot has more than 30 hours 

(1) if pilot has less than 20 hours 

73. Ml: Merit score on Scenario #1: Ml = (Cl) x (El). 

74. M2; Merit score on Scenario #2: M2 = (C2) x (E2). 

75. M3; Merit score on Scenario #3: M3 = (C3) x (E3). 

76. M4: Merit score on Scenario #4: M4 = (C4) x (E4). 

77. MECH: Mechunic: (0) = not a mechanic, (1) = mechanic. 

78. PROPCONl: Proportion of control movements to inquiries in Scenario #1: 

PROPCONl = CNTRLl/ll 

79. rROPCON2: Proportion of control movements to inquiries in Scenario #2: 

PROPCON2 = CNTRL2/I2 

80. PROPCON3: Proportion of control movements to inquiries in Scenario #3: 

PROPCON3 = CNTRL3/I3 

81. PROPCON4: F>roportlon of control movements to inquiries in Scenario #4: 

PROPCON4 = CNTRL4/I4 

82. PROPCONT; Proportion of total control movements tc total Inquiries in all 

four scenarios; PROPCONT “ CNTRLTOT/TOTINQ 
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83. RAT: Substitute variable for RATING used to plot initial data tables. Takes 

on same values as RATING. 

84. RATING: Rating type - 

1 = Private 

2 = Commercial 

3 = Air Transport 

85. RATSCORE: Variable dividing ratings into two groups - 

0 if private pilots (RATING = 1) 

1 if commercial or air transport pilot (RATING = 2 or 3) 

86. RECENCY: Relative amount of flying time in pasi year - 

1 = more than 50 hours 

2 = letween 20 and 50 i’ours 

3 = less than 20 hours 

87. S: Specific subjects involved in the GAT e^qjeriment - 

0 for subject numbers 11, 31, 32, 33 

1 for subject numbers 28, 34, 35, 38 

88. SEHRS: Hours of flying in a single engine aircraft. 

89. SEHRSLOG: Natural logorithm of single engine flying hours; 

SEHRSLOG = LOGj; (SEHRS) 

90. SHRSRANK: Variable designating upper and lower qiiartiles for single 

engine hours: 

0 if SEHRS se 488.75 

1 if SEHRS 22075.25 

91. SUB: Variable dividing subjects - 

0 if subject number is ^ 30 

1 if subject number is > 30 

92. SUBJECT: Subject number (N = 40) 

93. T: Variable designating upper and lower divisions for the variable TIM: 

0 if TIM < .625 

1 if TIM > 1 

94. TC: Variable designating upper and lower quartiles of TOTCOR; 

0 if TOTCOR < 10 

1 if TOTCOR 2 17 

95. TDELTAC: Sum of the changes in subjective criticality estimates for all 

four scenarios; TDELTAC = TCRITAFT - TCRITPEF 
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96. T£: Variable designating the upper and lower quartiles of TOTEFF; 

0 If TOTEFF s 42 

1 if TOTEFF 259 

97. THBSLOG: Natural logarithm of totaly flying hours; 

THRSLOG * LOGj; (TOTHRS) 

98. THRSRANK: Variable designating upper and lower quartiles for total 

fl]Ting hours; 

0 if TOTHRS ^ 1007 

1 if TOTHRS 2 5375 

99. TIM: Time attribute of an alternate airport - fl 3 dng time to the airport 

100. TM : Variable designating upper and lower quartiles for total merit; 

0 if total merit ^ 129.25 

1 if total merit 2 235 

101. TOTCOR: Total correct score for all four scenarios; TOTCOR = Cl * C2 + C3 + C4 

(possible correct = 0-20). 

102. TOTCRITAFT: Total of subjective criticality estimates for all four scenarios 

after being provided with the answers; 

TCRITAFT = CAl + CA2 + CA3 + CA4 

103. TCRITBEF: Total of subjective criticality estimates for all four scenarios 

before oeing provided with the answers; 

TCRITBEF = CBl + CB2 + CB3 • CB4 

104. TOTEFF: Total efficiency score for all four scenarios; 

TOTEFF = El + E2 + E3 + E4 

105. TOTHRS: Total flying hours . 

106. TOTINQ: Total number of inquiries for all four scenarios; 

TOTINQ = II " 12 13 + 14 

107. TOTMERIT: Total merit score for all four scenarios; 

TCmvTERIT = Ml + M2 + M3 + M4 

108. TOTTRAKS: Total number of tracks for all four scenarios; 

TOTTRAKS = TTl + TT2 + FT3 + TT4 

109. TOTUTRKS: Total number of unique tracks for all four scenarios; 

TOTUTRKS = UTl + UT2 ^ UTS + UT4 
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110. 

111 . 

112 . 

113. 

114. 

115. 

116. 

117. 

118. 

119. 

120 . 
121 . 

122 . 

123. 

124. 
1^5 . 
126. 


TRA: Variable used to plot the TRAINING values in the data tables; 

1 * military 

2 = civilian 

TRAINING: Type of training (military or civilian). 

TTl: Total number of tracks in Scenario #1. 

TT2: Total number of tracks in Scenario #2. 

TTS: Total number of tracks in Scenario #3. 

TT4: Total number of tracks in Scenario #4. 

UTl: Number of unique tracks in Scenario #1. 

UT2: Number of unique tracks in Scenario #2. 

UTS: Number of unique tracks in Scenario #3. 

UT4: Number of unique tracks in Scenario #4. 

WX; Weather attribute of an alternate airport; includes ceilings and visibilities. 

YCXtngOLD: Variable designating the upper and lower divisions of the age 
category; 

0 if age • 30 

1 if »*ge > 50 

Zl: Ratio of correctness to total tracks for Scenario #1; 

Z1 = Cl/TTl. 

Z2; Ratio of correctress to total tracks for Scenario #2; 

Z2 = C2/TT2. 

Z3: Ratio of correctness to total tracks for Scenario #3; 

Z3 = C3/TT3. 

Zl: Ratio of correctness to total tracks for Scenario #4; 

Z4 = C4/TT4. 

ZT: Ratio of total correct to total number of tracks for all four scenarios; 

ZT “ (Cl + C2 - C- ^ C4)/(TT1 + TT2 + TTS + TT4) 
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drive gear failure) do demonstrate the monotonlc relationahtp to rating one 
would expect. Scenario 4 (frozen static port) shows mixed results. 

Five subjects hold aircraft mechanic licenses (A and P) as well as pilot 
licenses. In terms of gross measures, the mechanics' performances are 
superior to the other groups. They have the top scores for knowledge, total 
correct, total merit and total efficiency. The only inconsistency again shows 
up in scenario 4 (frozen static port). However, since that problem relates 
to S3rmptoms more likely to be directly observed in their role as pilot rather 
than mechanic that result is not totally unexpected. More extensive analysis 
of die diagnostic performance data will follow in section K. 

E. Decision Making Phase of P 'P Scenarios 
Procedure 

The decision making phase of the paper and pencil exercise was divided 
into two parts: an information seeking part and a rank ordering of altemati\*es. 
The goal of the study was to determine the type of decision rule a pilot would 
use in a given problem, and to determine his worth structure concerning die 
characteristics of airports to which he might divert if it became necessary. 

The decision making portion of the experiment was begun after the pilot had 
completed all four scenarios in the diagnosis phase described above. 

One basic scenario was used throu^out the decision making {^ase and is 
given in .Appendix H. The mission of the hypothetical flight was to fly from 
Bangor, Maine, to Glens Falls, New York, for a business meeting. The 
flight was to be made in a Cherokee Arrow and the weather at the time of the 
flight, both along the route and at the destination, was marginal. Though there 
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IT HIDPOIHT TOTAL MERIT SCORE 



Table VI-5 


Diagnosis Performance Means by Rating 



Total Population 

PVT 

COMM 

ATP 

A & P 

Score on knowledge svirvey 

12.5 

10.5 

12.3 

14.1 

14.6 

Correct on Scenario 1 

2.7 

3.2 

2.8 

2.0 

3.4 

Correct on Scenario 2 

2.9 

2.7 

2.9 

3.0 

3.0 

Correct on Scenario 3 

3.4 

2.2 

3.4 

4.1 

4.0 

Correct on Scenario 4 

4.0 

4.1 

3.8 

4.3 

4.0 

Total correct 

13.0 

12.3 

12.9 

13.4 

14.4 

Total merit 

176.0 

156.0 

179.0 

186.0 

203.0 

Total efficiency 

50.0 

45.0 

51.0 

50.0 

53.0 
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was no severe weather forecast (In the form of thunderstorms, turbulence, 
or ice), the prevailing rainy and drizzly conditions required the fli^t to be 
conducted under Instrument Flight Rules (IFR). In fact, weather conditions 
were such that the fli^t would be in instrument meteorological conditions 
for almost the duration of the time aloft. 

After the pilot was given a brief introduction on the mission of the flight, 
the navigation chart of the area, and the airplane, attention was turned to 
analyzing the weather in detail and filing a flight plan. The pertinent 
weather information was given to the pilot in a text WTltten in ordinary 
English. This text is given in Appendix H. The wording of the text was 
intended to reproduce what one would normally he ir in a telephone conversa- 
tion with a weather briefer. All of the weather information needed to plan 
the flight was included. After the pilot confirmed that he had read and under- 
stood all of the weather, the next step was to compute and file a fli^t plan. 

In order to save time, the flight plan in Appendix H had already been com- 
piled based on the reported weather, and was shown to the pilot for his approval. 
The most Important features of the flight plan were reviewed by tlie pilot, 
including flight routing, cruising altitude, and estimated time enrc te. 

After reviewing all of the information on the airplane, weather, and fli^t 
plan, the pilot was asked if he would normally attempt a fli^t under the stated 
conditions. He was also asked if there was any other information he would 
like to have concerning the proposed flight. His responses to these questions 
were recorded, and it was then time to embark on the flight. 
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TTie description of the flight started with a routine lift-off from the Bangor 
Airport at the planned departure time. Climb-out after departure and the 
transition to cruising flight was uneventful. The pilot followed the progress 
of his flight on a very simplified enroute navigation chart (see Appendix H), 
which portrayed the intended route of flight, radio navigation aids and fixes, 
and the departure and destination airports. 

The flight continued uneventfully until a point about midway along the route. 
At that point the aircraft encountered a serious problem with its electrical 
system. The problem was investigated (in the text) and was determined 
to be an inoperative alternator. After the problem was clearly defined, an 
upper time limit estimate was provided to indicate how long the aircraft’s 
systems could rely on the reserve electrical power of the battery. 

The section of the text in which the problem was introduced and discussed 
contained several key pieces of information for the pilot. First, the symptoms 
and the diagnosis set the stage for the need to divert. The straightforward 
statement of the diagnosis was intended to give each pilot, basically, the 
same perception of the pro'?lem. This was of great importance since the 
focus of this part of the paper and pencil exercises was on the decision issues 
rather than diagnosis. If left to their own diagnostic devices, it would have 
been unlikely that all pilots would have perceived the problem In the same way. 
Ne.xt, the ramifications of the problem were clearly assessed. Having only 
battery power left to run electrical equipment, the problem was urgent in terms 
of time. The consequences of flying beyond the lifetime of the battery were 
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serious; the flight would be trapped aloft with no means of communication 
or navigational guidance. Finally, an estimate of the degree of time urgency 
was given when the estimated maximum time the battery would be useable was 
stated as being not longer than fifty minutes . 

The paper and pencil scenario had now reached the point where the pilot 
was called upon to use his own personal decision skills in the problem. The 
first task was to conduct an information seurch on the attributes of potential 
diversion airports. This included ceiling, visibility, navaids, terrain, 
availability of radar and distance and heading to the diversion airport. The 
purpose of this task was to determine the search strategy and decision rule 
the pilot used in shopping for an airport to which to divert. The second task 
involved ranking a group of sixteen airports from "most preferable" to 
'least preferable" based on their attributes. 

The Information Seeking Task 

In this task the pilot was required to search for an airport to which to 
divert. The pilot was s plied with Figure ni (In Appendix H) which 
portr.. .ed all the airports in the area. (He was cautioned that all the airports 
shown should not be assumed to be within his range in terms of battery time. ) 
As he viewed the new chart, the pilot was read the Instructions given in 
Appendix H. The experimenter was to act as the air traffic controller and 
would provide the pilot with the information he requested. The information the 
e.xperlmenter was prepared to give was summarized in Table A4 and shown to 
the pilot. 
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In all there were sixteen potential diversion airports and the experimenter 
had six pieces of information (from four questions) on each. The total store 
of information maintained by the experimenter is given in Appendix H. 

Each pilot was told he had two minutes to conduct his search and to select 
an airport to which to divert. The mention of the two minute time limit was 
intended to place a sense of time urgency on the problem, but it was not 
enforced. In most cases, however, the pilot had finished his search and 
selected an alternate airport before the two minute limit had expired. 

The experimenter provided the pilot w'ith each piece of information that the 
pilot requested. The experimenter recorded the sequence in which the 
information was requested. The pilot continued to request information 
until he had found an airport and revealed his choice to the experimenter. 

At this point the information seeking task was completed - 

F. The Ranking of Alternatives (Decision Phase) 

Information from ranking of alternatives was used to interpret the informa- 
tion seeking phase. In this phase the pilot was asked to rank sixteen alternative 
airports from "most preferable" to "least preferable" given his problem situation. 
Each airport was described in terms of four attributes, namely, air traffic 
control (.ATC) services at the airport, the weather at the airport, the time to 
fly from present position to the airport, and the best instrument approach 
facilities there. These attributes were chosen because they were independent 
with respect to each other, and also because they were the more pertinent 
items to consider in this situation. 
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Each attribute was varied over two levels: a "hig^" value (in terms of 
pilot preferability) and a 'low value". (For example, weather lOOO/S vs. 

500^ 1. ) All possible combinations of hi^ and low attribute levels resulted 
in a total of sixteen alternatives to be considered. The end product was a 

2*^ full- factorial design as shown in Table VI-6. 

Each alternative airport was depicted on a 3 x 5 inch card in terms of 
the four attributes. The sixteen cards were shuffled ,prior to the experiment) 
and laid out before the subject in a random fashion while the experimenter 
vead these instructions; 

"I have a set of cards here; each card describes an airport in 
terms of ATC services, weather, the flight time from your 
present position to the airport, and the approach facilities 
there. I would like you to rank these airports from your "most 
preferable" to 'least preferable", given the situation you are 
in. Recall that you have, at the very most, fifty minutes of 
battery time left. You may find it useful to divide the airports 
into "subgroups”, rank the airports in each subgroup, and then 
reconnect the subgroups as appropriate. Afterw-ards, make a 
final check of your rank and adjust it as you think necessary. " 

Subjects were given as much time as they needed to complete the ranking 
task, but rarely did it take longer than five minutes. While performing the 
task, subjects generally appeared quite involved and made meticulous 
adjustments to the rank before yielding a final ordering. When the subjects 
had completed the ranking task and were satisfied with their final prcduct, 
the experimenter recorded the ==equence and the ranking task was complete. 

In an attempt to estimate how "real" this simulation seemed to the pilot, 
and to determine the pilot's relative risk taking tendencies, 'he experimenter 
posed a series of questions for the pilot to consider. The questions asked 
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Table VI- 6. 2** Factorial Layout of Airports 
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the pilot how far he would go down his list of ranked airports to find one 
with maintenance facilities to repair his airplane — a question which seeks 
to find the limit of his diversion options. (The line of questioning used is 
contained under the Go- No-Go Instructions of Appendix H. ) ^Iten the pilot 
finished these questions both the diagnosis and decision making sections of 
the paper and pencil e* '‘rcises w’ere completed, and the pilot was invited to 
participate in other events at the workshop. 

.Analysis and Results 

The analysis proceeded first with the ranks provided by the subjects in 
the ranking task and then with the information seeking data. The first part 
of the analysis was aimed at modeling the pilot's worth function and determin- 
ing if worth functions are related to pilot background variables. The theory 
of conjoint measurement was used to model the worth functions (Kr«u,iz and 
TN’ersky, 1971). The second part of the analysis centered a the informa- 
tion search and how it related to worth functions. The analysis wb.ch follows 
was performed on the first 29 subjects . 

Results of Ranking Task 

A list of the ranks made by the subjects is given in Table VI-7. In this 
table the "name" of the airport refers to the airport with the same dimensional 
cont .guration as shown in Table VI-6. It should be mentioned that the airport 
name was not revealed to the pilot during the experiment in order to prevent 
any biasing effects that may have resulted. The numbers in Table VI- 7 
correspond to the positions in the rank that the airports were assigned by the 
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subject. The convention was adopted that sixteen equals ’’most preferable". 

It is evident that most of the pilots agreed airport A w'as most preferable 
and airport P was least preferable. However, much variation is seen in 
the airports in between. 

The additive model in Equation (1) below was assumed to be the under- 
lying psychological process in the worth structures and was proven to be 
the correct choice throu^ a series of axiomatic tests performed on the 
ranked data. (The ranked data of Subject 4 did not conform to the tests and 
his data was dropped from further analyses. In effect, the subject showed 
no logical preferred order. A 65 year old retired pilot, he may not have 
understood the instructions.) 

W(Xz) = Bate • ATCj - • WX^ * . TIM^ - B^pp • APP^ (1) 

where is the psychological worth of airport z, and .^TC^, WX^, TIM^’ 

and APPz independent variables describing airport z in terms of 

•AT. services, weather, time and approach aids respectively. The 

ir dependent variables took on a value of -*-l for Ae high level or -1 for Ae 
lo V level of each attribute. The "B "-coefficients are Ae "weights" each 
subject assigned to a certain attribute in his ranking scheme. The B-coefficients 
w e obtained by performing a regression analysis where Ae rank position of 
airport z was substiAted (according to conjoint measurement) for W(X^). The 
resulting coefficients are shown in Table VI-8. The range of values for Ae 
coefficients is 0.250 to 4.000. .An interpretation can be offered if one con- 
siders all four coefficients for each subject. The coefficients for Subject 1, 
for example, are 1.000, 2.000, 4.000, and 0.500 for ATC, weaAer, time. 
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Table VI-7. List of Airport Ranks by Subject 
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Table VI- 8. Coefficients of flie Additive Wordi Function 


(SUBJECT 4 DELETED. 24 SUBJECTS RCnAIlliriC) 


SUBJECT 

1 

2 

3 

5 

6 

7 

8 
9 

le 

11 

12 

13 

14 

15 

16 
17 
10 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 


BSUBATC 

1.000 

2.000 

0 . 500 

1. 125 
0.500 

4.000 

1.250 

2.500 

2.500 
4 . VOO 

2.000 

1.625 
0.625 

1.875 

1.000 

2.375 
3-250 

1.250 

1.875 

2.000 

3.500 

2.000 
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0.875 
0.875 
0.750 
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2.000 

0.750 
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1.500 
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0.750 

0. 750 
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4.000 
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4.000 
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4.600 
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0.7.50 
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1.375 
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3.300 
0.500 
0.730 
0.500 
0.250 
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0.500 
0.375 
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0.625 
0.500 
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1.000 
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1.875 
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1.375 
0.625 
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and approach, respectively. This can be interpreted as follows: the 
worth of having the time value at the hi^ level (15 minutes) was twice that 
of having the weather value at the high level (1000 feet ceiling and three miles 
visibility), four times the worth of having ATC at the high level (tower with 
radar), and eight times the worth of having the approach at the high level 
(ILS). In other words, the most important feature about each airport 
for Subject 1 was time followed by weather, ATC, and approach, respectively. 

In order to test for the validity of the additivity assumption in the model, 
the preference ranks determined from equation A for the sixteen airports 
were plotted against the original preferences for each subject. Spearman 
rank correlation was computed to estimate the fit of the model derived ranks 
with the actual ranks. Figure \T-8 depicts a typical plot. The correlation 
coefficients ranged from 8.74 to 1.00 for the 29 subjects indicating the model 
additivity w'as an acceptable assumption. 

The next step was to determine if any relationship existed between the 
worth function coefficients and pilot background variables. Since no measure 
of performance exists in this experiment, the data was examined to find 
relationships or explain differences. The basic approach was to dichotc...ize 
the sample population based on several different descriptors of a pilot's 
background and skill. The means of the coefficients for the resulting two 
groups were then compared to see if any significant differences occurred as 
a result of the division. The divisions were performed on the basis of flight 
experience, training, tjrpe of pilot certificate, ty^pe of flying most commonly 
done, and measures of ability determined by the knowledge survey and other 
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Figure VI-7 
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means. A summary or :oefficient means by pilot category is provided in 
Tab’"* Vl-9 along with the critei la used to split the sample population. 
Significant differences (at the p •< . 10 level) are enclosed in dashed lines. 

Analysis of Worth Functions By Subjects and Groups of Subjects 

The basic approach in this section is to dichotomize the sample popula- 
tion based on several different descriptors of a pilot's background and skill. 

The splits are performed on the basis of flight experience, training, tsrpe 
of pilot certificates, type of flying most commonly ei^aged in, and measures 
of ability determined by the knowledge survey and other means. The worth 
coefficients of Table Vl-9 become the center of attention in this analysis. 

This analysis covers the first thirty subjects used in this test. 

The first dichotomization is performed on the basis of total flight exper- 
ience in terms of fli^t hours. A bar graph representing the distribution of 
total hours is shown in Figure VI-9. As evident in this graph, the distribution 
of total hours is in no way "normal", and the criterion used to split the sample 
is somewhat arbitrary. However, the sample was split at the natural break 
nearest the 50th percentile. Because the sample is more heavily loaded 
with experienced pilots (many people consider pilots with SOO or more hours 
to be "experienced") the search for the natural break in total fli^t time pro- 
ceeded from the mean toward the "inexperienced" end. The criterion used 
to split the pilots was 1100 hours. Nine pilots were in the lower category 
and 21 pilots in the hi^er category. A t-test was performed to determine 
if there were any significant differences between the means of the B coefficients 
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Table Vl-0. Summary of Coefficient Means by Pilot Category 
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*Slgnlf leant differences enclosed In dashed lines. 







Table VI-9 (con't. 
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*Slgnlflcant differences enclosed In dashed lines. 



Figure VI-8. Distribution of Total Flight Hours of Sample Population 
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for the two groups. At the p ~ . 10 level, no significant differences were 
found. 

Splits of the sample were made based on the number of Instrument 
Fll^t Rules (IFR) hours and sinple engine airplane hour®. A bar graph 
of the distributions of each are shown in Figure VI- 10 and VI- 11, 
respectively. As in the case of total flight hours, these distributions are 
far from normal, and the "cut" was made in the same manner. At the . 10 
level, no Sie,nificant differences were found. 

The tjT>e of training a pilot received was used as a criterion to split 
the sample. There were seven military trained pilots and twenty three civil 
trained pilots. A t-test was performed on the worth coefficients anr* a 
difference which was significant at the . 10 level (p = .06) was observed for 
the mean value of (Recall that is a measure of the importance of 

air traffic control facilities in airport worth evaluation. ) For civil trained 
pilots the mean value of B„._ was 1.92 and for military trained pilots it 
was 1.25. 

There are several possible interpretations of this difference, but most 
allude to the pilot's attitude toward ATC facilities which are formed by pre- 
vious exposure. In military pilot training programs, much more emphasis 
is placed on emergency procedures and resolving in-flight problems than 
in civil pilot training. This may lead to an attitude of greater self-reliance 
in problem sifnations on the part of the military pilot, and a reluctance to let 
too much of the problem "out of t!ie cockpit". Additionally, military trained 
pilots may feel more strongly that .ATC facilities would be of only U nited 
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Distribution of IFR Hours of Sample Population 
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Figure Vl-30. Distribution of Single Engine Hours of Sample Population 



96 


IBM BBM 3RM 4BM tSM BM« TBM MM tSM 

• IRCUE BHCIIU: mOM 



value in this scenario. Admittedly, if battery power had been depleted and 
no communications were possible, ATC services would be of no use at all. 

The pilot sample was split on die basis of the type of certificate the 
pilot held. In this case, the twenty one pilots with Private and Commercial 
certificates made up one group, and the eight pilots with Airline Transport 
Pilot certificates made up the other group. The split was made in this fashion 
because the differences between private pilots with instrument ratings and 
commercial pilots are not great. The experience and proficiency require- 
ments for the issuance of those certificates are nearly the same. In many 
ways, the commercial pilot training and certification process provides only 
slight extensions of skill to private pilots with instrument ratii^s . On the 
other hand, the stringent eligibility and proficiency requirements for the 
issuance of the Airline Transport Pilot (ATP) certificate have led to the feeling 
that ATP airmen are the ’’cream of the crop". The difference between the 
two groups of pilots in terms of tested ability is distinct. 

Some notable differences were observed when comparing the worth 
coefficients of these two groups. The mean value of a measure of the 
Importance of weather to a pilot in this situation, was 2.49 for private and 
commercial pilots and 1.48 for airline transport pilots. A t-test was performed 
and this difference was found to be significant (p = .05). Another difference, 
significant at the . 10 level, was observed for th^ value of a measure of 

the Importance of the time attribute. For airline transport pilots B^.^^ had a 
mean value of 2.53 and for private and commercial pilots the mean value was 
1.56. 
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Tlie difference in between the two groups is most likely a function 
of training and the relative 'level of preparedness" to fly in adv*'rse weather. 
Althou^ pilots in both groups are trained in the procedures and maneuvers 
to be used when flying in bad weather, airline transport pilots are required 
to perform those maneuvers to much greater accuracy on fli^t tests. Also, 
in meeting the greater experience requirements for the ATP certificate, 
airline transport pilots have been exposed to more poor weather situations 
than their private and commercial pilot counterparts. In summary, airline 
transport pilots have reason to feel more confident about their flying skills 
in relation to marginal weather. 

Some interesting comments can be made about the difference in mean 
values of for the two groups. Because the mean value of vras 
higher for airline transport pilots (2.53) than for private and commercial 
pilots (1.36), one mi^t casually suggest that airline transport pilots are 
rrore cautious. The higher value of time could be interpreted as a desire 
of airline transport pilots to avoid flirting with the problem by landing quickly. 
This is the opposite of what one would expect, especially in view of the 
supposedly "stress hardening" experiences an airline transport pilot faces 
in his training and career. However, a more probable explanation for the 
observed difference is that airline transport pilots can take advantage of their 
skills to consider more airports. They may feel better prepared to conduct 
fli^t operations into an airport which is nearby but has poorer weather, and 
can Jierefore take advantage of time. Private and commercial pilots, though, 
may feel that some airports, even though they are close by, are bej'ond the 
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limits of their skills in terms of weather and facilities. Hence, they sacrifice 
time for better conditions and services. 

The type of flying most commonly done was also used as a basis to divide 
the pilot sample. Pilots who engaged primarily in business, light commercial, 
or pleasure flying made up one group. Pilots who were involved with airline 
or military flying comprised the other group. The spli' was made in this 
fashion because the hi^ly structured environments in which airline and 
military pilots operate are similar in many ways. They are both usually 
required to fly in and out of busy terminals and heed schedules, policies, 
and other disciplines. Pilots who fly for business, light commercial, or 
pleasure concerns, however, operate in a much more relaxed atmospiiere 
and dictate their own policies. Based on this split of the sample population, 
a significant difference (p = .024) was observed for the coefficient 
The mean value for business, li^t commercial, and pleasure flyers was 
1.96, while the value for airline and military pilots was 0.90. 

An explanation of this difference could be that military and airline pilots 
fly much more frequently in congested areas, and are mostly under the sur- 
veillance of an ATC facility. Given this day-after-day exposure to ATC, they 
are more aware of its abilities and limitations. Another potential explanation 
is the same one noted earlier when contrasting civil trained and military trained 
pilots. Because of the intensive initial and recurrent training in normal and 
emergency operations they receive, military and airline pilots may wish to 
solve in-flight problems with "on-board" resources rather than let too much 
of the problem outside of the cockpit. 
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All of the previous comparisons were based on classifications of pilot 
experience. There were, however, four measures of pilot ability and self- 
evaluation that were used to classify the pilots as well. The following dis- 
cussion treats comparisons made on the basis of the pilot's knowledge of 
aircraft systems, his ability to diagnose problems in fli^t, and estimates 
of the perceived risk he assigned to the problem. 

The knowledge survey which was administered to the subjects before the 
ejq>eriment was designed to estimate their knowledge of aircraft systems. 

The mean score was 12.3 out of a maximum of 20, and the distribution of 
scores was approximately normal, as seen in Figure VI- 12. The division 
w'as made at the mean and no significant differences were found in the worth 
coefficients of the resulting two groups at the . 10 leve’ . 

The pilot's diagnostic ability was estimated in the first half of the decision 
making workshop. In the four diagnosis scenarios, pilots were scored on the 
closeness of their diagnoses to the real problems, and these scores were 
summed up to yield a total correct score. The split was made at the 50th 
percentile of the total correct score which ranged from 5 to 20. At the . 10 
level significant differences were observed for two coefficients. Pilots in 
the lower half of the sample based on the total correct score had a mean value 
of 2.20 for the Bj^pp coefficient, while pilots in the upper half had a mean 
value of 1.37 (B^pp is a measure of pilot worth for the instrument approach 
dimension, and, for this difference, p = .080). A possible interpretation of 
this difference is that diagnostic ability parallels a pilot's perception of his 
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Figure VI-11. Distribution of Knowledge Survey Scores 
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SCORE OR KROVLEDGE TEST 



flying skills. The NDB approach, aside from being less accurate, requires 
more headwork and skill than an ILS approach. 

The other coefficient in which a . 10 level significant difference was 
observed is (p = . 076). Pilots in the upper half of the sample had a 
mean value of 2.25 while those in the lower half had a mean value of 1.37. 

This difference may again relate to the perceived level of skill. Pilots 
hi^er in diagnostic ability may not perceive the problem situation to be 
any more time critical than pilots in the lower half, but they can take 
advantage of closer airports more often. They believe they have the skills 
necessary to meet the challenges of poorer conditions which may accompany 
the closer airports. The notion that flying skills and diagnostic skills are 
related should be examined in future research. 

During the preflight preparation stage of the decision making scenario, 
each pilot-subject was asked if he would normally attempt the flight under the 
stated conditions. Ten of the pilots indicated they would not try it while 
20 said they would. A significant difference (p = .026) was observed between 
the two groups for the coefficient for approach aids, B^pp* Pilots who said 
they would not attempt the flight had a lower mean value for Bj^pp (1.01) than 
those who said they would try the flight (2. 11). This observed difference does 
not lend itself to a simple, straightforward interpretation. One would expect 
the relatively cautious pilots who would not attempt the fli^t to prefer the 
better approach aid (in this case the ILS) so they could have more in their 
favor. However, if one looks at the other differences in coefficient means 
even though they are not significant, some Insight is gained. Pilots who would 
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not go on the flight had a hi^er value for time, and a lower value for weather. 
This trend leads one to believe that the leery pilots do not wish to push their 
luck in terms of time. Hence, they put more wei^t in the time factor and 
take emphasis away from other factors. 

The final dichotomization of pilots was performed on the basis of their 
responses to the questions on maintenance facilities. In essence, each 
pilot was asked how far down the list of airports, arranged from most pre- 
ferable to least preferable, he would go to find the necessary maintenance 
facilities to repair his plane. Fifteen pilots said 1, 2, or 3 airports and 
fifteen pilots said 4 or more (the range 1 to 14). The mean value of 

was 2.28 for pilots who responded with 3 or less (call them "less risky" pilots) 
while the same measure for the (more risky) pilots who responded with 4 or 
more airports was 1.40. This was significant at the . 10 level (p = .083). 

This difference can be attributed to conservatism of the pilots in the less 
risky group. In the same manner they are reluctant to take risks by "passing 
up" too many preferable airports, they are unwilling to pass over a closer 
airport (in terms of time). 

A summary of the results of all the analyses performed in this section is 
given in Table VI- 10. It is interesting to note that significant differences In 
worth function coefficients were not a result of flight differences, rather, were 
related to the grade of pilot certificate, the amount and type of initial and 
recurrent training, and the tj-pe of flying most commonly done. This suggests 
that training and repeated exposure to testing situations are the variables which 
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can predict the general form of a pilot's worth function. A closer examination 
of the training and certification process is in order. 

Once a pilot obtains a private pilot certificate with an instrument rating, 
there is very little he must legally do to continue exercising the privileges of 
his certificate. He can continue to accrue many hours of flight time but he 
is required to demonstrate, on only a sporadic basis, that he is maintaining 
his basic skills. Airline transport pilots and those pilots who fly for the 
military or airlines, however, must maintain a higher level of skill regardless 
of the amount of flight time they have. Many are required to demonstrate 
proficiency in all sorts of demanding situations and at much more frequent 
intervals than the biennial flight reviews required of general aviation pilots. 

The general "level of preparedness" is much higher for military and airline 
flyers than for the rest of the flying population. All of this lends support to 
the notion that the total amount of fli^t experience is not as important as 
the amount and quality of initial and recurrent training in determining the 
general worth structure of a pilot. 

G. Results of Information Seeking Task 

Pilots were referred to the simplified charts of Appendix H when performing 
information searches. Because of the hypothesized strong winds aloft (out of 
the southwest at 30 kts. ), airports which were closer in terms of distance 
were not always closer in terms of time. Table VI- 10 shows rank orderings 
of the f irports, from nearest to furthest, in terms of both time and distance. 
Table VI- 11 depicts a subject by airport listing of worth values. 
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Table VI- 10. Rank Orderings of Airports In 
Information Search From Nearest to Furthest 


In Terms of Time 


1 

E 

(12 minutes) 

I 

(23 miles) 

2 

I 

(13) 

E 

(34) 

3 

H 

(18) 

0 

(40) 

4 

N 

(19) 

N 

(40) 

5 

P 

(20) 

P 

(41) 

6 

A 

(22) 

F 

(43) 

7 

J 

(22) 

H 

(49) 

8 

F 

(22) 

J 

(54) 

9 

O 

(22) 

G 

(54) 

10 

L 

(23) 

B 

(54) 

11 

D 

(25) 

A 

(58) 

12 

C 

(26) 

L 

(63) 

13 

M 

(28) 

C 

(63) 

14 

G 

(28) 

D 

(65) 

15 

B 

(31) 

K 

(68) 

16 

K 

(32) 

M 

(69) 
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All twenty-nine subjects considered airport I (i.e.. Information was 
requested about I) and twenty-nine considered airports E and K . The 
frequency of consideration decreases for airports further away. The 
number of times each airport was chosen as an airport to which to divert 
in this scenario is given in Table VI- 12. The most popular choice was 
airport N. 

E^ch pilot's search pattern was analysed in an attempt to determine the 
search and decision logic used when seeking information. Though many pilots 
displayed definite search patterns, determination of a decision rule was not 
possible for two reasons. The first is that many pilots reverted to standard 
ATC infori.iation exchange formats. In requesting weather information, for 
example, they would always req-»est "ceiling" before "visibility" as in the 
format for such data in weather reports . The second reason is that many 
pilots estimated bearings and distances directly from the chart, and hence 
they had information for which no record of request was made . 

Some interesting observations can be made, however, when a comparison 
is made between the results of the ranking task and the Information search 
task. Table VI- 13 lists the pilots whose most important attributes were 
ATC, time, weather, and approach aids, respectively. 

Airports N and J were the only ones chosen in the group of pilots who 
considered "time" to be the most important attribute. Airports N and J are 
in the top half of the airports ranked according to closeness in terms of time. 
Pilots who considered other attributes to be the most important, however, 
chose a much broader range of airports . Pilots in the group who considered 
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Table VI- 12. The dumber of Times Each Airport 
Was Chosen as a Diversion Airport 


Airport 


Number of Pilots 


N 

J 

O 

C 

A 

P 

E 

M 


10 

8 

3 

3 

2 

2 

1 

_1 

30 


lOf 



Table VI- 13. Most Important Attribute for Pilots 
(Subject Numbers Shown) 


ATC 

Weather 

Time 

Approach 

S7 

S8 

SI 

S5 

S9 

SIO 

S2 

S6 

Sll 

S13 

S3 

S12 

S17 

S14 

S16 

S24 

S18 

S21 

S17 

S26 

S22 

S23 

S19 

S27 


S25 

S20 



S28 




S29 




S30 
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weather to be the most Important attribute chose airports A.C, J, M,N,0, P; 
pilots who thought ATC to be most Important chose airports O, N, J, E, and 
pilots who thought Approach Aids were most Important chose airports 
A,C,J, N. The same trends were apparent in the airports considered by 
pilots in each group. The general interpretation is that pilots who placed 
most emphasis on '‘time" did not venture as far to find a suitable airport 
as did pilots with other priorities. Pilots who placed emphasis on an attribute 
other than time, on the other hand, either were forced to search continually 
for airports with better conditions (and maybe further away), or felt that 
time was not a serious issue and searches of airports further away were 
feasible. 

H. Analysis of P & P Tests (All Phases) 

Since the two major phases of the P &• P tests yielded almost ninety 
measures on each subject (both raw data and derived measures) it became 
necessary to develop an overall analysis strateg\' to derive the maximum 
amount of useful information from such data. Note that the data was divided 
into four basic categories: The Diagnostic Phase Data, the Decision Phase 
Data, the Subject Biographical Data and Knowledge Data. Those data bases 
were individually analyzed for descriptive statistics and derived measures of 
pciiormance. 

In the diagnostic phase the raw data included (for each scenario and for all 
scenarios) merit, correctness score, efficiency, total inquiries, total unique 
tracks, total tracks, criticality assessments before and after the diagnosis, use 
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of control inputs and derived measun (correct score/total tracks) and 
CORINQT (correct score/total inquiries). These variables are defined in 
the glossary. 

For the diagnostic phase, means and variances of these measures both 
for each subject across the four scenarios and for all subject across each 
scenario were reported in the Master Data Table (Table VI- 14). In addition, 
the distribution of these data was examined for outliers. Rank correlations 
were examined to find associative relationships (see Table \T-15). These 
data were then used in the combined data analysis w'hich is described below. 

The decision phase data included subject airport rankings, information 
seeking profiles, go/no go responses before the flight and the number of 
airports the subject was willing to pass to locate an aircraft mechanic. 

As described above, conjoint measurement analysis was employed on the 
ranking data to derive worth functions. These were expressed as pilot 
wei^tings for weather (WX), navigation aids (APP), radar services (ATC), 
and time to the airport (TIM) (see Glossary). The decision phase data were 
then used in the combined analysis. 

The third major data base involved subject biographical and knowledge 
data. These are shown in Table VI -14. These data were subjected to descrip- 
tive statistical analysis and transformations to adjust for outliers in distri- 
butions. Table VI- 14 describes the findings from the knowledge and biographical 
data. 

The main thrust was the combined analysis of diagnostic data, decision 
phase data, and knowledge and biographical data. Sections H, J, and K which 


111 



Tab’e VI -14. Master Data Table 
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d 

USHfiO«3C0 

CO 

CO 

d 

a 

d 

a 

9 

d 

a 

;;£HeS-Jd 

d 

9 

d 

a 

d 

a 

9 

a 

a 


9 

9 

9 

d 

9 

a 

9 

d 

9 


9 

9 

9 

9 

9 

9 

9 

9 

9 


N 

9 

9 


9 

9 

d 

9 
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e 
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3 
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d 
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9 

9 
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9 
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9 


to 

N 
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a 

N 
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9 
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•0 
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9 
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d 

a 
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i 
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a 


9 
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9 

d 
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CO 
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•• 


•• 
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a 
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9 

N 

9 
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9 

9 

a 

3 

9 

a 

a 

N 

d 


"• 
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•• 

"* 

“• 

d 

d 

pp 
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9 

9 

9 

mm 

d 

9 

a 

9 

N 

9 
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9 

a 

a 

a 

a 

9 

a 
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to 

N 

d 

a 

9 

a 


d 

a 


to 

9 

d 

9 

9 

_ 
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9 

p. 
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pp 

HOHHCC<^X 

N 

pp 

a 
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9 

d 

9 

N 

a 
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HOH-K9 

10 

9 

9 

d 

9 

9 

9 

a 

pp 

CO 

d 

a 

a 

d 

d 

a 

a 

9 
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d 

p. 

a 

d 

- 

d 

d 

d 

- 
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d 

d 

a 
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- 
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d 

a 

- 
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9 

N 

a 

3 

3 
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9 


9 

9 

9 

d 

9 

3 

Q 

9 

d 
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a 

a 

9 

9 

9 

N 

d 

9 








p« 



U9 

10 

a 

a 

a 

a 

a 

a 

a 

a 

DH9 

9 

9 

a 

a 

d 

a 

d 

a 

9 
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9 

9 

a 

d 

a 

d 
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a 


d 
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d 

a 

a 
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a 
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Table Vl-14. Master Data Table (con't.) 


NM 


»IO««9NIOOO 




Cl 

d 

- 

9 

9 


d 

9 

9 



$ 

9 

9 

9 

9 

9 

9 


$ 



S 

9 

9 

9 

9 

9 

9 

N 

9 



$ 

9 

9 

9 

9 

9 

9 

0 

9 


N ^ 

0 

9 
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9 

9 

9 

X 

9 



10 

9 

9 

9 

0 

9 

9 

d 

9 



• 











c« 

d 

d 

d 

Cl 

9 

d 

9 

9 



w 

- 

- 

9 

9 

- 

9 

N 

l> 



d 

9 


9 

9 

9 

9 

- 

d 


O^Uk9i 

0 

- 

9 

9 

9 

9 

9 

9 

9 
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0 
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9 

9 

9 

9 

9 

- 

0 
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9 

9 

9 

9 
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- 

9 

0 
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N 

d 

0 

** 

0 

N 
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0 
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« 
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d 
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9 

» 

0 

0 

9 

9 

N 

0 

N 
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X 

9 

9 

N 

X 

N 



CO 


9 

d 

9 


N 

d 

d 

c 


Cl 

d 

ci 

ci 

d 

ci 

n 

d 

d 
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0 

9 

9 

N 

9 

CO 

9 

9 

9 

E- 


0 

9 

9 

9 

9 

CO 

9 

9 

9 



0 

9 

9 

9 

9 

« 

9 

9 

0 

QD 
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0 

9 

9 

9 

rt 

9 

0 




K» 
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9 

9 

0 

rt 

9 

d 

X 



•1 


ci 

d 

d 

d 

X 

d 

M 

QD 













9 

9 

N 

9 

N 

9 

9 

9 

9 



9 

9 

9 

9 

9 

9 

9 

9 

9 

QD 
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9 

9 

9 
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9 
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9 

9 

9 

9 

0 

9 

9 

9 
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0 
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9 

9 

9 

0 

9 

9 
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C« 


C3 

CO 
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d 

X 

ci 

« 
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9 

9 

N 

9 

CO 

9 

9 

9 

9 
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9 

9 
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0 
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9 

0 

< 
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9 

9 

9 

N 

CO 

0 

9 

9 

N 



9 

9 

9 

Q 
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d 

9 

9 

X 

K 



• 


• 


• 

• 

* 

• 
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d 

d 

•M 

ci 

Cl 


X 


< 
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9 

9 

9 

9 

9 

9 


9 
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9 

9 

9 

9 

9 

9 

-p 

9 

pj 


9 

9 

9 

9 

9 

9 

9 

N 

9 



9 

9 

9 

9 

9 

9 

9 

fi 

9 

< 
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9 

9 

9 

9 

9 

X 

9 

U 


rt 

N 

d 

CO 

CO 

ci 

9 

- 

9 

Pi4 













» 

9 

M 

9 

N 

9 


d 

9 

H 


9 

9 

N 

0 


9 

O' 


X 




9 

9 

N 

X 

9 

d 

X 

9 

QD 


N 
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9 
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0 

9 

d 
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9 
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d 

0 

Cl 

d 

d 

H 
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9 

9 
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9 

9 
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< 


CC 

9 

N 

9 

9 

9 

X 

d 

X 

H 


W 

9 

9 

9 

9 

•• 

X 

d 

X 



CO 

9 

9 

9 

9 

N 

X 

d 

X 

X 


CO 

9 

9 

0 

9 

0 

X 

d 

X 



rs 

9 

9 


9 

C5 

X 

d 

X 



« 

9 

«. 

CO 

9 

d 

X 

d 

"" 



9 

9 

9 

9* 

9 

9 

9 

9 

9 



9 

9 

s 

9 

X 

9 


9 

9 



9 

9 


9 

M 

9 

N 

9 

9 



N 

9 

i 

9 

X 

9 

0 

9 

9 



9 

9 


9 

— 

9 

X 

9 

9 


QpAoo«;;ckco 

CO 

9 

i 

9 

X 

0 

d 

9 

9 



d 

0 


0 


N 


0 

0 



9 

9 

9 

9 

9 

9 

9 

9 

9 



9 

9 

9 

9 


9 

9 

9 

N 



9 

9 

9 

9 


0 

9 

0 

Cl 



9 

9 

0 

9 

N 

0 

9 

9 

N 



9 

9 

9 

9 

0 

0 

9 

0 

CJ 


&.e;o&-;^rKci 

9 

9 

0 

9 

X 

0 

9 

0 

N 



0 

9 

d 

d 

CJ 

0 

9 

Cl 

Cl 



9 

9 

9 

9 

9 

9 

9 

9 

9 



9 

9 

9 

N 

9 

9 

9 

>C 

9 




9 

9 

9 

9 

9 

9 

't 

'f 



9 

9 

0 

9 

9 

9 

9 

X 

9 



9 

9 

9 

9 

9 

9 

9 

X 

9 


0 . 3 : 00 * ;^CK- 

9 

9 

9 

9 

9 

9 

9 

0 

9 



9 

9 

9 

9 

9 

0 

9 

“ 

9 



9 

9 

9 

9* 

*0 

9 

9* 

9 

9 


VH 


N « 


^ LO « 




9 

9 

- 

• 

• 

• 

9 

9 

• 

• 

- 


- 

• 

• 

• 

b« *" 

9 

9 

9 

9 

9 

- 

- 

- 

XdX 9 

9 

9 

9 

9 

9 

9 

9 

9 

>*exKoc.Jx 

• 

• 


w4 

9 

• 

• 

- 


- 


- 

9 

- 

•H 


9 

HBXo()X<ae;OiC 9 

• 

- 

• 

• 

• 

• 

• 

9 


0 

9 

- 

9 

- 

9 

9 

9 

^0. ^ 

9 

- 

9 

9 

- 

9 

- 

M 

H - 

- 

- 

9 

- 

- 

• 

9 

9 

xBXxX<aE;^ 9 

• 


9 

• 

9 

• 


• 

HU 

- 


■ 

- 

- 

• 


9 

HU 

- 


. 

9 

• 

- 


9 

HB * 

- 


• 

• 

• 

• 


9 

X 

• 


• 

• 

• 

• 


• 

HCWUH<«U 9 
1 

9 

1 

9 

9 

1 

9 

T 

X 

1 

d 

\ 

9 

XU.JH<U9 9 

1 

9 

9 

9 

9 

9 

1 

9 


9 

9 

9 

9 

9 

9 

9 

9 

XU.JH<UX 1 


1 







XUpJH<Ud 9 

1 

9 

9 

9 

9 

9 

d 

9 

DUpJH<U-« 9 

d 

1 

9 

1 

9 

1 

9 

♦ 

1 

X 

X 

1 

9 


X 


9 

X 

X 

9 

9 

_ 

N 

X 

9 

0 

d 

d 

9 

X 

0 

0 

52 

9 

N 

X 

9 

9 

N 

9 


c: 

d 


9 

9 

9 

0 


UOX-^B9r' d 

9 

0 


9 

N 

9 


d 

9 

N 

X 

9 

X 

0 

0 

X 











9 

9 

9 

9 

9 

9 

9 

9 

9 

N 

9 

N 

9 

9 

0 

X 

9 

X 

9 

d 

9 

9 

0 

Cl 

X 

0 

X 


9 

9 

0 

9 


X 

0 

X 




N 

9 

wm 

2 

0 

X 

9 

N 

9 

X 

9 

N 

X 

0 

X 

9 

9 

9 

9 

- 

9 

9 

9 

9 

9 

X 

9 

X 

9 

9 


9 

9 

9 

X 

9 

X 

9 

9 


9 

9 


X 

9 

X 

9 

9 

N 

9 

9 

UOX^BOX 9 

2 

9 

2 

0 

0 

0 

0 

9 

9 

X 

9 

CO 

9 

d 

X 

d 

9 

9 

9 

9 

9 

9 


9 


9* 

9 

9 

9 
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9 

9 

9 

9 
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9 

9 

9 

X 

9 

0 
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9 

® 
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9 
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d 
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9 
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9 

0 
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9 

90 

- 

9* 

9 

9 

9 

9* 

9 

9* 

X 

9 

N 

N 

X 

9 

X 

9 

9 

X 

*■ 

9 

9 

X 

9 

X 

'f 

9 

X 
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9 

9 

X 

9 

X 

X 

9 

X 

0 

9 

9 

X 

9 

X 

X 

9 

UOX-B9- X 

X 

9 

9 


9 

X 

0 

9 

X 

CJ 

9 

9 

X 

9 

X 
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9 

9 

9 

9 

9 

9 

9 

9 

9 

0 

0 

X 

X 

9 

9 


3 

X 
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9 

9 

9 

9 

9 

C5 


N 


0 

X 


9 

9 

B 

0 

N 
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9 

d 
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9 

N 

X 
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9 
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N 
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0 

- 
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9 
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« 
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9 

9 
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0 
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0 
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0 
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9 
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Cl 

9 

CJ 


Cl 

9 
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d 
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9 
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9 

N 
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Table VI- 14. Master Data Table (con’t.) 
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- 
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9 

10 

N 

9 

9 
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-• 

3 

3 

3 
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3 

- 

- 

3 



ao 

O 
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9 

ao 

9 

9 

3 
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3 
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3 
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3 

3 

N 



o 
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9 
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9 


9 

CD 











































V030V0 

w« 

9 

mm 

mm 

mm 
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mm 


HO 

N 

N 

9 


9 

CD 

9 

3 

N 















O 

O 
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QO 

10 

9 

10 

10 

10 

ao 

9 

3 

3 


• 

W 

• 
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• 

• 

• 

• 

• 

• 
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C9 

o 

Cl 

Cl 

9 

9 

9 

3 



3 
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• 

3 

* 

• 

• 

• 
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• 

« 
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>0 
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o 

o 

9 

9 
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10 
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9 
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- 


O' 

N 
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9 

9 
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N 


9 

9 

9 
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• 

• 

• 

• 

• 

• 

• 

• 

• 
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S 

• 
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9 

9 

ao 

n 

9 

9 



3 
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9 

9 

mm 
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9 










>0 


9 

9 

CO 

CD 

9 

3 

9 
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. 

3 

• 

• 

• 

. 

• 

• 

• 


HO 

•m4 

■* 






Cl 




CD 









VO 

10 

10 

9 

9 

9 

ao 

ao 

3 

3 



9 



















9<H^3 

• 


• 

• 

• 

« 

• 

• 

• 

s 

DHO 

o 

o 

C9 

9 

r 

CD 

9 

3 

3 











u 
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o 

10 

10 

9 

9 

9 

9 

3 

3 



9 









«o 

GO 


9 



CO 

3 

3 
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«« 







9 








QD 
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>0 

9 

- 

C9 

CD 

9 

Cl 

3 


9 

- 

9 

9 

9 

9 

9 
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9 
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9 

9 

9 

N 

3 

3 

3 


3 

3 

3 

3 
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3 

3 

3 

QQ 
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0 

9 

9 

9 

3 

9 

9 

9 
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3 
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9 

9 

1- 
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VO 
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3 

3 


3 
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9 
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9 

mm 
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ao 
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S 

ao 
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9 

9 
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9 
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9 
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• 
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3 

9 
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9 
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•" 
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3 
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ao 
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T 

I 

M 
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3 
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follow detail the results of this combined analysis. Essentially uiree 
types of analysis were involved. First, Spearman Rank Correlations were 
examined for all data Inputs with emphasis on those correlations with major 
dependent measures. These included merit, efficiency, and correctness 
measures and their derivatives from the diagnostic phase and weighting 
functions from the conjoint measurement analysis of the decision phase. 

The rank correlations provided not only insight into two variable relations 
but indicated input variables for the stepwise regression analysis which 
followed . 

The stepwise regression analysis predicted dependent measures from 
the set of independent measures (biographical, knowledge measures and 
independent per'^ormance measures). 

Finally, partitions on the independent measures were examined to ascer- 
tain differences in performance measures. The splits varied depending on 
the nature of the variables in question, e.g. , pilots with aircraft mechanic 
(A and P) ratings vs. pilots without A and P ratings, differences in pilot 
ratings, etc. In addition, some performance measures such as total merit 
were split into top and bottom quartiles to ascertain differences in oth'r per- 
formance indices. The performance measures are listed and defined in the 
Glossary, Table Vl-3. 

I. A Rank Correlation Analysis - Combined Data 

Table VI- 15 depicts the Spearman rank order correlations for the major 
Independent and dependent variables in the analysis. T-tese correlation tests 
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were conducted as a first cut through the data prior to stepwise regression. 
The correlatioas enabled the researchers to get an overall view of relation- 
ships between the variables of interest. Clearly, some significant correla- 
tions resulted because the two variables were related not to each other but 
thr h a third variables. These effects were evident with the stepwise 
regression. 

Table \T-15 reports rank correlations for experience variables, knowledge 
scores and overall tests. Although a log transform of total and single engine 
hours was employed to adjust for the skewed distribution in these factors, 
such a transformation has no effect on rank correlation. The comments to 
follow consider a relationship to be significant if the a value is s. 10. 

It is interesting to note that total hours is related only to s ingle engine 
hours and not to any of the performance measures. Sii^le engine hours is 
related to knowledge scores. This is to be expected since the knowledge 
test and its subscores were based on single engine aircraft operation. The 
negative correlation of the decision factor, weather, with single engine hours 
suggests that high experience levels lead to less emphasis on weather in the 
diversion decision. 

Knowledge scores are highly related to total merit (a = .0001) - a findii^ 
that holds up in the entire analysis. Knowledge about aircraft subsystems 
definitely affects diagnostic performance in a positive manner- Knowledge is 
also related to total correctness ( a = .005) and total efficiency ( a = .003) 
both of which make up total merit scores. Hence, the knowledgeable pilot is 
more likely to not only get the right answers in diagnostic tests but also to get 


123 



BACraSROUND. DIACRDSI8. AlfD TOTALS 

SPEARHAN CORRELATION COEFF.CIENTO / PROU > IRI UNDER IIOtRUO*© /" NITHDER OP OBSERVATIONS 






Et* 

?? 

r^io® 

H'O® 


S 3-® 


b. 10019 

UNO 
H09 
pso • 

P •• 

9 


2 C5’^9 

— K ^ 

gES*1 


0«’^9 

C»01^ 


PNC9 
Ol9S« 
BIO Cl 

BO • 


< *9 

— .9 

9 -9 

W9 

U9 

QB 

W9 


PNC59 

N9N9 

9C«»9 

9^9^ 

— 099 

W — - 

B9® 

BN- 

H901 

r:c59 

U99 


Be • 

So • 

P .® 

O -9 

P * 9 

9 

W® 


S9-9 

PO-9 

QN99 

^99CD 

N«99 

B-09 

NN 

«® 


B9 

N9 

XN® 

01 • 

e • 

Po * 

•9 

•9 

< -9 

9 

9 

W9 


§ 099 
Nn©^ 
WO — 

< *9 
W9 


N9 — 9 
— ’♦9^ 
Bo9 
MC59 
So » 

P *9 
p9 


W09® 

390^ 


a N«-9 
099 
c- W-® 

QQ /2CS9 
fc* c-9 • 

P < -9 


N-99 
N9N9 
99 
-9 
9 • 
•9 
9 


— 099 
Bo 0 9 
U09 
^N9 

5-9 

W9 


R --*-9 
U B9 99 
W-9 
^ 009® 

9 Pn • 

B < .® 

S W9 


9099 
pOM9 
U09 
XN® 
B9 • 

= -9 

U9 

Cfi 


9e-9 

E-fl-9 


OW-9 

HN0I9 

9 

rt9-0 

SB99 


0--9 

99-9 

9N99 

P939 


CN99 

— 99 9 


BX99 

— 9«C5 


9999 

ae:e99 

»J0 9 

— 0^9 


m09 

BN9 


099 



U — o 

— 9e 

X99 

^99 


X9® 

UBCO 



99 


e99 

5-99 

BN • 

B9 . 


BN • 

s- . 


P N • 

9 • 


5n • 

99 • 

5 *9 

N *9 


£ *9 

H -9 

o 

< .® 

•9 


9 *9 

P *9 

u® 

B9 


Po 

9® 

X 

09 

9 


E9 

9 

X 

O 



P 





S 

I 

0999 

H9 — 9 


099® 

No99 


0999 

uefi9 


-999 

XNC5 9 

0999 

99C59 

¥ 

C999 

9999 


C99 9 

rf.9?:9 


B999 

PfiBrt 

^99 

B9N 

9 

^99 

K9CS 

s 

U99 

U99 

•M 

099 

B99 

X99 



X99 

-99 

u 

p^99 

— e® 

S£ 

X9 9 

9-9 

fl2® • 

BO * 


B® . 



X® . 

99 * 

O 

P® • 


P -9 

C *9 

£ 

S -9 

5 *9 

X 

C *9 

P -9 


< .® 

B*t® 


09 

B 

U — 

09 

9 

*5* 



O- 

— 9 


1 

U 

X 


B 

2 


< 


< 1 



X 



S 



o 




124 


CATSrU2 KNOWI.EIX; TOTMERIT ZT C01UNQ1’ TOTEFF TOT I NO TOTTRAKS TOTCOU CATBORt 

t.ouuue U.f>ai27 0.S4I6I e.4BNa4 e. 47336 e.43462 -0.40000 -0.42639 0.3S92S 0.324411 

O.UCOO O.OUOl 0.0003 0.0013 0.0020 0.0032 0.0036 0.0061 0.022H 0.0411 



BACKGROUND. UIAGNOSIS, AND TOTALS 

SreAIUlAN CORRELATION COEFFICIEN'I'S / PROD > IR( UNDER IIO:RlIO=® / NUMBER OF OBSERVATIONS 


NNCM’^' 


kM d ^ ^ 


naec39 


^ISC19 



©©-• 

»©9« 

aN© 

le© 

X©© 

X©© 

xa© 

C5 • 

Pso • 

P« • 

Pa * 

•© 

< •© 

< -© 

< • © 

© 

u© 

U© 

a© 

Hn©© 

ej — ©© 
S©©'«> 

©M^© 

aaa© 

©©e*^ 


a©a© 

K© — 

u— © 


a^© 

^©© 

9D©© 

|-©© 

xa© 

CS« . 

?«5 • 

w C? * 

p© • 

o •© 

< *© 

p .© 

< .© 

c;© 

U© 

© 

I 

a© 

— N’^'© 

XCd© 

xa©© 

aaa© 

=i-©^ 

SiTi^yt 


a©a^ 

u— — 

< — o 


u©© 

5BN© 

£©© 


jg© 

- 

po • 

r « • 


< .© 

p •© 

r- •© 

o •© 

V© 

?? 

2t 

g* 

c©©© 

o©-^© 

©a©© 

s®-^ 

cl©—® 

©N©^ 


©©a© 

J©- 

Ut9© 

ua — 

a©© 

niw© 

E^a© 

X©© 

Pa© 

S© • 

Sio . 

P« • 

p© • 

S -© 

o •© 

< .© 

P •© 

b2© 


u© 

© 

QQ 

S 

©«-•© 

ae©© 

XN-© 

C2© — 

15-©^ 

a©©<© 

Sa©© 

r-o® 

b3©© 

<a© 

pN© 

ti©© 

H^© 

c4a© 

£© • 

C© - 


p© • 

p -© 

P .© 

© •© 


g® 

© 

1 

s* 

g? 


c:sseo 

UC3-- 

i”® 

O 


auC5©^ 

g*^® 

© 


u©© 

H»© 

gt® 

© 


N«©^ 

©© 

©© 

© . 
•© 


B -«^0 


CN-© 


NlO©^ 


^©-© 


P-© 


a©© 


— © 


a©© 

U©© 


Pa© 


©o 


^©© 

F© • 


ba • 


a - 


ap • 

P •© 


P -o 


• © 

5 

© •© 

e© 


© 


© 

a 

a© 

a 1 



5 


X 





© 




c©©© 

5 

Pa-© 

r 

p©-© 

P 

P©— © 



s©©© 

X 

©flO© 

a 

-a©© 

p?i© 

a 

P© 


xa© 

a 

©M© 

a© 

X 

a© 

P 


b3 

ua© 

© • 


a • 

a 

aa • 


FN • 

• © 

P 

•© 


© •© 

a 

E •© 

© 


© 


a© 


p© 


s 


S 


Gb 

P 










o 



Ch 




aaa© 


aa-© 



Pa-© 

-©p© 

a 

pp-© 

sa ©© 


©©-© 

Ca©9 


“ 'C a a 


©a©© 

1.© ©a 


e©©© 

a Lft^ TT 


— p©© 

— Cl ©a 


© © 

— vft a a 


PP© 


:b©© 


a 

a — © 

ap- 


ao© 

-«i ^ 


-©© 

— P P 


-© 


^ n © 

©© 


— a© 

X ©© 

< 

© 

p© 

< 

5^9© 

a© 


« • 


Sp . 

Cl * 


fi© • 

pa • 


ca • 

Cl • 

u 


Cl • 


• © 


© •© 

•© 


© *© 

< .© 

P 

*p *© 

• o 


P •© 

*® 


o 



© 

1 

P 

a© 

a© 


p* 

© 

P 

© 

1 

© 


«©c© 


»©©© 

Cl©©© 


Poo© 

ca©© 


a©©© 

-aci© 


Sb©©0 

P©-© 


a- w© 


a©o© 

a -Cl© 

F 

-©Q© 

-p©a 


©©9© 

a©e© 


b.99© 

^T5©©© 


aci© 

2 

a©© 

a Cl© 

1-^ 

a©9 

P© — 


a©© 

aa© 


b:©C> 

r»© © 

a 

X?I© 


xc© 

XCl© 

a 

^©© 

P© 


P©o 

XP© 

Su 

P©© 

-:©© 

a 

pa • 


• 

pa • 


c® * 

a • 

o 

c© • 

PCI • 

b; 

©© - 

Fci - 

X 

< •© 


< •© 

< .© 


p •© 

*© 

a 

P '© 

< •© 

b. 

P •© 


P 

a© 

& 

a- 

a© 

P 

p- 

© 

(m 


a© 

p 


a© 



it 



p 

p 


p 



o 


a f 

a 


a 



P 



p 



p 




125 



SPEARNAH COmiELATlOn COEFFICIENTS / PROD > IRI UNDER IiO:lUIO»« / NUMBER OF OBSERVATIONS 


9QN9 


»» • 
c 




aioa 
< *o 

ttCD-® 
sao^ 
SiSna 
^NO 
>IO • 

c -a 


i®a® 

H©®© 

HN®© 

• aaa 

c®aa 

o®-a 

!—a 

Has 

xaN 

— ® 


i©e 

a • 

s ^ * 

tf® • 

• ® 

H -® 

o *® 

® 

k® 

u® 

1 



laa® 

pa®© 

a—®© 

!®sa 

H-aa 

©©aa 

IN® 

<©a 

p©® 

--® 

a® 

H®a 

a • 

♦ 

A M • 

• ® 

• ® 

H -® 

1® 

® 

® 


1 

1 




of N w 
aeaea 
-•-d 
C59“# 


r«‘'C 

? •• 


2^>ea^ 

^ N© ® 

Ha® 

P'^® 

® 


® ®e® 
cN®a 
— ®^ 
Ha® 

® 


®a®a 

.^aa 



b.®'-*® 

sa©® 

b.a®® 

r«® 

ua© 

He® 

H©© 

ON • 

p® • 

H •© 

H •© 

® 

® 


^® 

au^®^ 

g ^® 
®® 


ca©^ 


a-’-® 


Haa® 

9® Na 

saa 
^a® 
esa • 

O -® 
u® 



CSV- 

® © 

UN® 

0 • 

Sa • 

•® 

H •© 

® 

p® 

N-© 

&.®©© 

a©® 

A,a®a 

.© © 

<©- 

a© 

0 ® 

■a - 

a • 

-© 

• ® 

:® 

® 


QD®®© 

QQ© a© 

Sn® a 

H-sa 

< — a 

PeS 

pea 

©-- 

MM • 

^a - 

H -® 

S -® 

C® 

— ® 

H 1 

< 

p — ©© 

£®a© 

e®®a 

r^aoa 

ues 

H-N 

UNa 

a — 

5- • 

a • 

0 -® 

“® 


® 


uaa® 

Haaa 


Ha-o 


sa®® 

"©s 

a© 

©N© 

®© 

ue® 

© • 


• © 

h"^© 

© 

p® 


Hrt^® -fa© H^-© 

— ®o^ ®«®^ 

a-© Ea® 

wa© /j^© -®© 

£© • r-a • a® * 

H .® < .© © •© 

e© u© u® 


wa® ® 
©a»a 

bh©® 

ea® 

o •© 


a — © 
«^a 

3© 

u 

a©©© 
a - — a 
;ja© 
/:3© 

H3 — © 
s ® © a 

3© 

A 

s 

?l • 

z 

P® • 

a • 

a. 

• © 

M 

< •© 

*© 

:u 

© 

H 

0 ® 

© 

< 


5®© «Na© 
i®a ©»va 

© ^®a 


©®»^® 
c^® a 
jnn 
xa© 
ca . 

© «® 
:»© 

QD I 



H®® ® 

Eaa© 


H® ® ® 

a® ® ® 


£® ® © 

ua© © 


O®©® 

r"'® NW 


S©®® 

SNa® 


-©©a 

b.ee a 

H 

E®® 

r-a a 


®® 

^©a 


Ho® 

N IV 

© 

-©© 

e© 


o® 

xe® 


® © 

H ©© 

E 

E© . 

a • 


® • 

Ha • 


® * 

pci • 

M 

© •© 

•® 


• ® 

< ‘© 


•® 

H •© 


0 - 

© 


M 

0 ® 

E 

M 

© 

c 



H 






0 



S 



H 




» ® © © ei a © 

j©a ©a*^a 

; © w aa 

>© H-f 
> * © - • 

•® H -9 

• © 



QD 

K 

O 


&a 

s 

o 

&h 

O 


c 

c 

s 


c 

c 


o 

o 

a 


cs 

c: 


lO 

1-^ 

I 

> 

;; 

!H 

:3 


U 


\ 


I! 


u 

e 


ao 


2 = 
z — 
c 

< ^ 


ocs®^ 
© — o« 
«^N© 

M •© 
&2© 

QD 

HX«^ 

©xcm 

—e« 
c ^ 
c •© 
u© 

I 

H0C5^ 

NCXX 

^ — 

-© 

© 

I 



s:x>o© 

OOI^^ 

S§5 

p« • 

© 

I 

sSe:^^ 

320^© 
HX • 

< .© 
U© 

I 

c|^ — © 

KJ^O^ 

£3® 

< .© 
©© 

I 


H I 

zxo 

- 

O -9 
©9 
I 




9CI9(^ 



©a.*.© 



ZXCIX 



ZCI9^ 



r-xx 



r*'C9 



t“* C5 ** 



P ^9 



pci - 



c© . 



p .© 



P -9 



9 



9 


o 







P^X9 


S 

&.N«*9 


z 

9^-»X 


u 

j* X 9 ^ 


o 

P-N 


< 

{kX 9 


© 



s 

r-N9 



Scj . 


p 

99 . 


© 

P .© 



P -9 



Z9 


u. 

9 



© 


O 

1 

X » X Cl S' 

§ 

Z9 X9 

©tt«^ 


PN — 9 

25^xx 


^ X X X 

9x 9 X 


SCI 9^ 

X <-C 

■»■ 

P — X 

A X X 

> 

9 9 

P XNC 


-9 

^99 

< 

X9 

< p9 • 

m* 

• 

35 • 

u 

X • 

© f- • 9 

•* 

• 9 

9 -9 

p 

*9 

5 |9 

< 

9 

1 

i? 

p 

9 

1 


C©z© 

7-NnOC5 

e«9 
» • 

.© 

9 

I 

Zei99 

ex-^ 

©Cl Cl 
p9© 

p© - 
c* *9 

9 

1 

^9>C9 

X9X^ 

©99 

XX9 

- 

< *9 
©9 
I 



< 


P999 
5c9X 
<99 
99 
9 • 
•9 


iiCl^© 

a--X9 

<9X 


9 

I 


-o c 
P 


X999 

<99 
X9 9 
r© • 

127 


-9CJ© 
H X 9 
Cl- 
Cl • 
•9 
9 
I 

X9X© 

P — 

XC^?5 

CX9 

S *9 
^9 
< 


I 

X 

9 

O 

z 

du 

O 


I 

— 9 
— — O’^ 
XZ9 
»^9 
Z9 • 

P •© 
gf 


r9 

i® 


a.9-9 
ShC' 9 ^ 
MtJ-© 
H9 9 

?"^9 

9 

I 

9999 
Z90^ 
— 99 
?-99 




Xo 
;; -+ 

i£5 
< • 
©9 
I 

PCI 
~ 9 

£x, 

r- *4 


— 9 
9 ^ 


X9 

9 


PN^9 

NC19'«> 

X’^ 

Ci • 
•9 
9 
I 

P999 

Z90I 

SS"^ 

O *9 
©9 
i 

PX99 
^CIN^ 
X99 
ao " 
ZCl • 

P -9 


» 


XX99 

X-^XX 

CX99 

XXCI9 

©XZ9 

©’^X 

9 


O^Cl^f 

P9 — 


Q 

©»X 

&3XO 

bl09 

<-N 

• 

s 

X9CI 

-3»9 

c499 

99 

A 


P- . 

• 

• 

X • 

Z 


< .© 

C *9 

O -9 

•9 

© 

\ 

©9 

1 

£? 

5? 

9 

s 

i 

a— X9 

PXCI9 

Pci — 9 

fc.9- 


Z 

S0^-x 

^N99 

N999 

U.99 

© 

u 

<99 

X-9 

99 

^99 

< 


C9CI 

W99 

X9 

P99 

X 

© 

PCI • 

ze • 

9 • 

pci . 



P .© 

P -9 

*9 

P -9 


b« 

09 

09 

9 

9 


9CIX9 

Z9©^ 

psj 

9 


— 9 

Z9S9 

99 

— NXX 

9 

PC»9 

9 

^© 

> 

X . 

9 

*9 


9 

1 

-9 

©XX9 

99 

XX«9 

9 

:=:cici 

O 

Pno 
5x - 

9 

C »9 


5t 


X X 'S* 9 CJ 
XO'^V 


^2^9 
S-^ . 

5s® 

I 


Cl 9 


is' 

< -i 

A 

^ T 


r- 9 9 9 7: X 

P90^ ^X 
p99 
^99 
X9 • 

H -9 


99 

X'f 

9 


x:i 
P'?9 



Table VI- 15. Rank Correlation Table (con't. 
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the answers more efficiently, i.e. he uses few inquiries to get the answer. 

As might be expected, and CORINQT are also highly correlated to know- 
ledge scores since they are derived from total correctness. 

One major finding in the correlations (which holds up with subsequent 
analysis) is the positive relation of knowledge with time (TIM) ( a = .0056). This 
means that more knowledgeable pilots place high emphasis on "time to air- 
port" in the case of a destination diversion decision. With concern for 
possible additional complications, knowledgeable pilots want to get the 
aircraft on the ground at the earliest opportimity rather than proceed farther 
to better weather or facilities. 

As expected, total correctness is related to Zj., CORINQT, total merit, 
and individual scenario scores. Table VI— 15 also indicates pilots w'ho have 
high correctness scores use fewer tracks and fewer inquiries than those with 
lower scores (TOTTRKS, a. = .0146) (TOTINQ, a = .0214). This again 
supports the link between diagnostic correctness and efficiency of diagnosis. 

Use of control activation related inquiries was negatively related to 
knowledge ( a = .0258), to total efficiency ( a = .0601) and to time weights. 

This was somewhat surprising in light of the value of selected control input 
tests to find problem causes, e.g. , prop cycling to ascertain the locus of low' 
oil pressure readings in scenario 1. 

The decision weights show obvious negative intercorrelations with each 
other since one cannot have high weighting on more than one attribute. The 
time-knowledge relation described above is again noted. Airports passed 
to locate mechanics in the decision tests is negatively related to efficiency 
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and knowledge scores and positively to total tracks used. If airports passed 
represents a crude measure of risk then high risk acceptance pilots have 
inefficient diagnostic procedures and are less knowledgeable. 

J. Regression Analysis 

In order to ascertain what factors predict performance measures, a 
series of stepwise regressions were performed. These are listed in 
Table VI- 16. The table indicates the dependent variable, the R^, jj, and 
the significant predictor variables from three sets of independent variables - 
biographical and experience, knowledge, and other independent performance 
measures. Candidate predictor variables are indicated at the top of the 
table. The general strategy was to include all predictor (indepandent) 
variables, even those which could be related to each other, e.g. , DIFT 
(total tracks - total unique tracks) and total tracks and total unique tracks. 

The model then selected which of these added the most to the prediction. 

The model permitted no variable to be introduced if it had been derived from 
another significant predictor variable. 

On the other hand, no predictor variable which formed part of the 
dependent variable was allowed in the regression. For example, since 
Zj, = total correct/'total tracks, a predictor variable could not be Introduced 
which contained total tracks, e.g. , DIFT (DIFT = total tracks - total unique 
tracks). 

When correctness scores or. ndividual scenarios are examined, C1-C4, 
some interesting results appear. Efficiency is a good predictor of correctness 
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for scenarics 2, 3, and 4. Knowledge subscores are good predictors for 
Cl and C4 only. Experience variables (training and rating) show up in Cl 
prediction and (recency) in C3. These are far less significant than efficiency 
as predictors. Control use is clearly important in scenario 3 as might be 
expected (this was the rough running engine scenario). The number of 
inquiries per track is a positive predictor of correctness in scenarios 
2, 3, and 4. It seems evident that scenario 1 differs from the other three 
scenarios probably because of its unique nature, i.e. , it required the 
pilot to seek information about conditions inside die cockpit, but not on the 
panel. Weather wei^ting was a negative predictor for C2 and C4. 

Pilots with good total correctness scores are characterized by high 
efficiency, a low number of total inquiries, a low number of tracks, a 
low DIET (total tracks - total unique tracks) and thus a high number of 
inquiries per track. Knowledge is also a good rredictor. Note, no 
biographical and experience factor predicts total correctness. 

Individual scenario efficiency scores show little predictability from 
knowledge test scores. Efficiency on scenario 1 is related to experience 
vaviables, i.e., training, rating, and recency - again supporting earlier 
findings on Cl. Most of the efficiency predictors on the scenarios include 
total tracks (negatively) for Ej, E 2 . E^, and unique tracks for Eg. Interest- 
ingly enough, control activation is a good predictor (negative) for El, E2, 
an ' E4. 

For total efficiency prediction, total tracks, and total control movements 
are negatively significant. Total correctness is also a significant predictor, 

again linking good performance with efficient performance. 
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Final Summary of Stepwise Predictions 
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Negative B- Value 
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In general, efficiency Treasures led to better prediction than correctness 
scores (higher R squares). What was surprising was the absence of know- 
ledge, biographical and experience variables as significant predictors for total efficiency. 

For total merit prediction, the knowledge score is the best predictor 
( a = .0001). Later, when extreme scores on total merit are examined through 
t-tests, knowledge scores differentiate well between high and low merit scores. 

Total merit is also predicted by DIPT. Low DIFT j'leld high merit scores which 
suggest^ a nonrandom approach by high performance score pilots. Not sur- 
prising, and CORINQT were much like total merit since all three measures 
are functions of total correctness. Knowledge was a significant predictor for 
these performance measures. No experience variables w'ere significant. 

CORINQT was like merit. Zj revealed total unique tracks and total efficiency 
as predictors. 

Table VI- 16 also shows that knowledge scores are significant predictors of 
time weighting and airports passed in the decision phase. In both cases 
CATSCRl was a negative prediction. CATSCRl deals with engine and fuel 
systems. CATSCR2 (electrical and cockpit operations) appears to be more 
germaine to the destination diversion decision. As expected, pilots, who weight 
time as critical, pass few airports. Pilots who consider weather weightings 
as critical are willing to pass airports to locate a mechanic. Although the 
R squares were low, biographical and experience factors appear for weather 
and ATC weighting. These will be examined later in partition testing. 

Since knowledge has bearing on performance it was decided to seek pre- 
dictors of the total knowledge scores. Table VI- 16 reveals that type of flying, 
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training, and recency of flight all are significant predictors. Non-pleasure 
flying, high recent experience and military training yield higher knowledge 
scores. Both approach aid weightings and total tracks are {.cod predictors 
of knowledge in an inverse direction. 

K. Tests on Data Partitions 

Table VI- 17 shows a series of tests on extreme partitions of major 
independent and key dependent measures to ascertain if differences might 
exist in extreme cuts through the data as compared to regression of the entire 
data set. Test candidates for the partitions included twenty-one performa-ice 
variables. Two state variables are easier to examine in this framework, 
i.e., mechanics vs. non-mechanics, military vs. civilian training and 
go vs. no-go preferences for the decision flight. Twentv dependent 
measures shown in Table VI- 16, plus two experience measures were considered 
for performance differe'^ces in the data splits. 

Pilots with recent flight experience (over fifty hours in the past year) 
were more knowledgeable and used less control activations than pilots with 
less than twenty hours in the past year. Surprisingly enough, diagnostic 
and decision performance were not different - perhaps due to the small 
number of low recency pilots . 

Total hours of experience ( < 1007 hrs. vs. > 5375 hrs. ) showed no 
relation to performance. Separating single engine hours experience revealed 
greater knowledge for pilots with over 2075 single engine hours vs. those 
with less than 488 hours. Splits on ratings revealed commercial and ATP 
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rated pilots have more knowledge than private pilots. For training 
partitions , military training leu to higher knowledge scores than non- 
military training. A split on IFR hours revealed a slight totul correctness 
difference with the two groups , < 175 hrs. vs. > 700 hrs. ). m all of die 
splits above there was little or no performance difference. 

When pilots under age thirty are compared to those over fifty, performance 
differences begin to appear. Younger pilots have hi^er merit, Zj and CORINQT 
scores than older pilots, yet have less experience (total hrs. and single engine 
hrs. ). Type of flying also showed performance effects. Pleasure flying 
pilots showed less knowledge, less experience, less efficie.icy, more tracks 
and larger DIFT than pilots who for airline, comm, olal, business, or 
military purposes . 

When performance measures are split to get profiles of high score pilots 
vs. low score pilots, many other Independent performance effects are noted. 
Focus is on dependent variables or other measures not related to the parti- 
tioner variable. When partitioning on knowledge, the high knowledge scores 

(5] 6) are associated with: 

a) higher single engine hours experience 

b) higher weightings on time in the decision phase 

c) higher correctness and merit scores 

d) higher efficiency scores 

e) fewer inquiries 

f ) fewer tracks 

g) fewer uruque tracks 

than pilots with low knowledge scores (less than 10). 
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DATA PARTITIONS ON TEST VARIABLES - CATSCRl (.0869) (0) means recency differences on CATSCRl 

BKKiRAPHICAL AND EXPEPIENCE was significant at = .0869. 'fhe (0) means .-igh recency group had higher mean 



CORINQT (.0943) (0) 
ATC (.0335) (1) 
TIIRSLOG (.0140) (1) 
SEIIRSLOG (.0198) (1 
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Summary of T-Tests 
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\Mien the top and bottom quartile on merit and correctness scores are 
examined, an interesting pattern emerges . Members of the upper quartile 
of performance scores are associated with: 

a) more knowledge 

b) greater weights or time 

c) fewer inquiries 

d) fewer tracks 

e) fewer unique tracks 

f) lower DIFT 

g) higher efficiency 

h) hi^er single engine hours (total correctness only and here 

a = .07) 

WTien upper and lower quartiles in total efficiency are examined much the 
same patterns as indicated above result. 

Since time weighting appears to be associated with good diagnostic per- 
formance, splits on time 'veights were made to test for performance effects. 
High time weightings look again like the list above, i.e. , greater knowledge, 
efficiency, correctness and merit with fewer tracks, fewer inquiries and lower 
DIFT. 

In the regression analysis, airports passed was predicted by knowledge 
subscores, time weightii^, and DIFT. Table VI- 17 shows that a split on 
airports (2 s vs. > 2) shows that low number of airports passed is related 
to hitler efficiency and knowledge and lower values for DIFT, unique and total 
tracks and total inquiries. 
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L. GAT Subjects in P/P Tests 

Eight of the subjects who participated in the FMS experiments also 
participated in the P/p experiments. All of the subjects from the first 
GAT scenario (fuel loss) and all from the second GAT scenario (partial 
power failure) participated. None from the third GAT scenario (partial 
navigation system failure) participated. 

Subjects 11, 33, 34, and 35 were used in the fuel loss FMS experiment. 
Subjects 28, 31, 32, and 38 were used in the partial power loss FMS exper- 
iment. 

Some tipicai performance measures for this group of subjects are high- 
lighted in Table \T-1S. The encouraging thing about this comparison is that 
the results appear to be reasonably consistent. The subjects for each FMS 
experiment are equally distributed with respect to airman certificates held 
(one Pv't. , two comm. , and one ATP in each group) in similar portions to 
the entire P/P group. 

Each GAT subject was ranked from one (best performance) to four 
(poorest performance) by a subjective evaluation of the experimenter present 
during all of the G.4T runs. Each subject was ranked separately for "Aviating", 
"Navigating", and "Communicating" for the scenario in which he participated. 
".Aviating" reflected basic stick and rudder skills. "Navigating" reflected the 
abilit>’ to follow the flight planned course and the subjects awareness of location 
along the route. "Communicating" reflected the professional nature of the 
information exchange bet\veen pilot and controller. All G.AT subjects took 
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Table VI- 18. GAT Subjects iVp performance 
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the open form knowledge survey prior to the experiment. These data are 
also noted in Table VI- 18. 

The first observatic i to make is that knowledge scores on both forms of 
knowledge survey seem to agree. Those scoring high on the open-form also 
score high on the closed- form. 

The second observation to make is that GAT performance rank is generally 
consistent with total correct (TOTCOR) and total merit (TOTMERIT) scores 
on the p/p scenarios. In GAT scenario 2 the rank order of "Aviating" in 
exactly the same as the rank order of TOTMERIT. Other rankings are less 
perfect but still exhibit the same general trends. 

Although the sample size is too small to draw definitive conclusions, taken 
as a group, results for these eight subjects seem to indicate that either FMS 
or p/p scenario experiments can be used to evaluate pilots with some 
assurance that relative rankings will be preserved. 

M. Summary 

The above analyses have found the same pattern running through the data. 
The three statistical approaches lead to the same conclusions. TTiese are: 

1. ) Correctness in diagnostic scores is highly related to efficiency 

in reaching diagnostic answers. 

2. ) Most biographical and experience variables do not appear to be 

related to diagnostic performance measures (the exception is the 
A & P mechanic rating which does appear to be related to 
diagnostic performance). 


144 



3. ) Knowledge scores are positively related to both correctness and 

efficiency in diagnostic performance. 

4. ) Of the four airport attribute weightings only high time weighting 

is related to high diagnostic performance. 

5. ) Patterns of information seekii^ for high diagnostic performance, 

i.e., high efficiency and high correctness, involve 

a) a minimal number of tracks employed 

b) a minimal number of unique tracks employed 

c) a minimal number of total inquiries 

d) a small number of track repeats, i.e. , DIPT (total tracks - 
total imique tracks) is small. 

Although most experience and biographical factors failed to be significant 
in the analysis, some did appear to be related to knowledge scores - such as 
high total single engine hours, high recency, military training background and 
non-pleasure flying. 
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VII. ANALYSIS OF PROCEDURAL COMPLIANCE 


A . Background 

At the same time that the GAT and paper and pencil scenarios were 
being developed for single pilot CIFE's, a parallel study directed toward 
airline cockpit crew operations was undertaken. The person responsible for 
this effort was Lt. Col. Jeffrey Schofield who performed the research as 
his Ph.D. dissertation project. A copy of that dissertation, "Aircrew 
Compliance With Standard Operating Procedures As A Component of Airline 
Safety", is on file as part of this project's records at NASA-Ames. 

Schofield used data generated in an experiment conducted in 1976 by 
Dr. H. P. Ruffell Smith under the auspices of the NASA-Ames Research Center. 
The Ruffell Smith research utilized a full-mission simulation to study the per- 
formance of fully qualified airline crews under varying conditions of workload. 
The cockpit was that of a Boeing 747 which accomodated the usual three-person 
crew plus two observers, a simulator operator /traffic controller, and an 
audio coordinator. The full-mission scenario used was built around a charter 
flight from Dulles .Airport to Heathrow' Airport (London) with a thirty-minute 
intermediate stop at Kennedy .Airport (New York) for fuel and cargo. The first 
segment placed relatively low workload on the crews, while the second seg- 
ment was much higher due to pre-programmed mechanical failures. 

Ruffell Smith concentrated on crew errors during the second segment (high 
workload) of the scenario. Ht was interested in establishing statistically 
significant physiological or historical predictors of crew’ performance during 
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the second leg. Schofield, on the other hand, chose to emphasize the routine 
or customary tasks of flight operations as exemplified by the first segment 
of the Ruffell Smith scenario. Furthermore, he was concerned with: 

1) quantifying routine crew procedures, 2) analyzing observed crew errors 
to identify which particular crew members were the primary causes of such 
errors, and 3) comparing measures of procedural compliance and operator 
error. 

The primary data used by Schofield came from the audio tracks of the 
FM tapes and handwritten documents generated by the Ruffell Smith study. 

This information was supplemented by data which was culled from the Aircraft 
Operating Manual, the Company Operations Manual, the Federal Aviation Regu- 
lations, crew handbooks, and assorted navigational documents. 

B. Procedures 

A procedure is defined as "a symbolic and mnemonic representation of a 
set of sensory, cognitive, and/or motor activities which, when recalled and 
executed within determinable tolerances, complete a task as designed''. 

The word "procedure" and its many aliases appear throughout aviation litera- 
ture. Schofield identifies nineteen separate words and phrases associated with 
aircrew operations which have procedural conotation. 

Schofield enume>'ates a set of normal operating procedures, as opposed to 
.Abnormal, Alternate, Irregular, or Emergency procedures, which represent 
an idealized sequence based on the events in the Dulles-JFK segment of the 
Ruffell Smith e.xperimental scenario. All of his procedures are considered 
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mandatory for normal flight operations in instrument meteorological conditions. 
Each procedure is identified by published format and the cockpit crew mem- 
bers expected to exhibit active procedural behavior. These are catalogued 
in Table VII- 1. 

The astonishing fact in this list is that 97 normal operating procedures 
can be identified for standard cockpit activities lasting approximately 75 
minutes. This lengthy list does not include any "optional" procedures or 
emergency type procedures. They represent only standard operating proced- 
ures for the first leg of the simulated flight scenario. 

Schofield has identified several empirical taxonomies which seek to 
classify these procedures in ways to identify useful relationships among 
them. One such grouping is the set of 21 crew coordination procedures shown 
in Table VII-2. Crew coordination procedures are emphasized since they cap- 
ture the essential ingredients of group leadership, crew management and 
behavioral conformity. Schofield examines the realtionships between meticu- 
lous compliance with coordination procedures and the crew errors noted by 
Ruffell Smith. 

C. Compliance Assessment 

Although Rufi.ll Smith used eighteen crews in his experiment, the quality 
of the data generated and the obsei vers were not the same for all ei^teen 
simulation runs. Schofield selected ten runs, which had the same set of 
observers and usable audio data throughout, for detailed procedural analysis. 
The 21 crew coordination procedures were further subdivided into checklists, 
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Table Vn-1 


NORMAL OPERATING PROCEDURES 


Index 

Number 

Name 

Format^^^Operator^^^ 
codes codes 

1. 

Basic ATC communications practices 

N 

PNF 

2, 

Preflight Radio Checklist 

C,N 

P2 

3. 

Gear pin status report 

N 

PI • 

4. 

Hydraulic system pressurization 

N 

PI & FE 

5. 

ATIS report 

N 

U 

6. 

Clearance Delivery coimimni cat ions 

N 

PNF 

7. 

Ground Control communications 

N 

PNF 

8. 

Pre-start Checklist 

C,N 

A 

9. 

Ground crew report 

N 

PI 

10. 

Cabin report 

N 

PI 

11. 

Engineer's Start Checklist 

C,N 

FE 

n. 

Start Checklist 

C.N 

A 

13. 

Engine starting 

N 

PI & FE 

14. 

Ground connections and hand signals 




report 

N 

PI 

15. 

Engineer's Taxi Checklist 

C,N 

FE 

16. 

Pre-taxi Checklist 

C.N 

A 

17. 

Transfer of EGT monitor 

N 

PI & FE 

18. 

Ground Control connnunications 

N 

PNF 

19. 

Taxi 

N 

PF 

20. 

Takeoff and departure briefing 

N 

PF 

21. 

Final weight and balance computation 

G.N 

FE 

22. 

Taxi Checklist 

C.N 

A 

23. 

Tower communications 

N 

PNF 

24. 

Passenger pre-takeoff announcement 

N 

PF 

25. 

Engineer’s Takeoff Checklist 

C.G.N 

FE 

26. 

Runway line up 

N 

PF 

27. 

Takeoff Checklist 

C.N 

A 

28. 

Thrust setting (takeoff power) 

G.N 

PF & FE 

29. 

Takeoff 

N 

PF 

30. 

Takeoff callouts 

N 

PNF 

31. 

Noise abatement departure 

G.N 

PF 

32. 

Gear retraction 

N 

PF & PNF 

33. 

Departure Control communications 




(initial contact) 

N 

PNF 

34. 

Thrust setting (rated power) 

G.N 

Pi & FE 

35. 

Departure Control communication 




(radar vector) 

N 

PNF 

36. 

Flap retraction 

G.N 

PF & PNF 

37. 

Altitude callout 

N 

PNF 

38. 

Internvedlate level off 

N 

PP 

39. 

Departure Control communications 




(climb clearance) 

N 

PNF 
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Table VII- 1 (con't.) 




- - - ■— (If 

(2) 

Index 


Format Operatrr 

Number 

Name 

codes 

codes 


40. 

Airways navigation practices 

N 

PF & PWF 

41. 

Thrust setting (rated power) 

G,N 

PF i FE 

42. 

Climb (below 10,000 feet MSL) 

K 

PF 

43. 

CoiT'^any departure report 

N 

U 

44. 

ARTCC communications (initial contact) 

N 

FNF 

45. 

Seat belt sign 

N 

U 

46. 

Climb (above 10,000 feet MSL) 

N 

PF 

47. 

After Takecff Checklist 

C,N 

PF & FE 

48. 

Altimeter reset [not applicable for 




cruising below 18,000 feet] 

N 

A 

49. 

^RTCC communications (route clearance) 

N 

PNF 

50. 

Cruise data 

G,N 

FE 

51. 

Altitude callout 

N 

PNF 

52. 

Level off 

N 

PF 

53. 

Mach number/airspeed crosscheck 

N 

FE 

54. 

Cruise 

N 

PF 

55. 

ARTCC communications (radar vector) 

N 

PNF 

56. 

Turbulence penetration 

N 

A 

57. 

ARTCC communications (radar vector) 

N 

PNF 

58. 

Turbulence exit 

N 

A 

59. 

ARTCC communications (route clearance) 

N 

PNF 

60. 

Fuel systems management 

C,S 

FE 

61. 

ARTCC communications (center change; 




initial contact) 

N 

PNF 

62. 

ATIS report 

N 

U 

63. 

Company arrival report 

N 

U 

64. 

^proach briefing 

H 

PP 

65. 

ARTCC communications (sector change,; 




initial contact) 

N 

PNF 

66. 

Approach data and speed bugs 

G,N 

A 

67. 

Passenger arrival announcement 

N 

PF 

68. 

Descent Checklist 

C,N 

A 

69. 

ARTCC communications (desct=nt clearance) 

N 

PNF 

70. 

Descent (above 10,000 feet M5L) 

N 

PF 

71. 

Altimeter reset 

N 

A 

72. 

Seat belt sign and landing lights 

N 

U 

73. 

Descent (below 10,000 feet MSL) 

N 

PF 

74. 

Approach Control communications 




(initial contact; clearance) 

N 

PNt 

75. 

Approach Checklist 

C,N 

A 

76. 

Category I Instrument Landing System 




(ILS) Approach 

G,N 

Pi- 

77. 

Approach radio checks 

N 

PF 6 PNF 
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Taole VII- 1 (con't. ) 


Index 

Number 

Name 

Forma^^^ Operator^ 
codes codes 

78. 

Altitude callout 

N 

PNF 

79. 

No smoking b^gn 

N 

U 

80. 

Approach Control communications 
(radar vector) 

N 

PNF 

81. 

Approach flap extension 

G,N 

PF & PNF 

82. 

Course bar and glide slope callouts 

N 

PNF 

83. 

Approach Control communications 
(approach clearance) 

N 

PNF 

84. 

Landing gear/landing flap extension 

G,N 

PF & PNF 

85. 

Landing Checklist 

C,N 

A 

86. 

Final approach fix (FAF) communications 

N 

PNF 

87. 

IaF irst.unent crosscheck 

N 

PNF 

88. 

Precision approach callout 

N 

PNF 

89. 

Ou* scan and visibility callouts 

N 

PNF 

90. 


N 

PF 

91. 

Landing roll callouts 

N 

PNF & FE 

92. 

Tower comraunlcaticns 

N 

PNF 

93. 

After Landing Checklist 

C,N 

A 

94. 

Taxi 

N 

PF 

95. 

Ground Control communications 

N 

PNF 

96. 

Parking 

N 

PF 

97, 

Blocks Checklist 

C,N 

A 


vl) Format Codes: C (Checklist), G (Graphical), N (Narrative) 

(2) Operator Codes: A (All), PI (Captain), PZ (Copilot), FE (Flight Engineer), 
PF (Pilot Flying). PNF (Pilot Not Flying), U (Unspecified) 
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Table Vn-2 


CREW COORDINATION PROCEDURES 
(To be used for quantitative compliance assessments) 


Index 

Letter 

A. 

B. 

C. 

D. 

E. 

F. 

G. 

H. 

I. 

J. 

K. 

L. 

M. 

N. 

O . 

P. 

Q. 

R. 

s. 

T. 

U. 


Procedure Name 

Pre-start Checklist 

Start Checklist 

Pre-taxi Checklist 

Transfer of EGT Monitor 

Taxi Checklist 

Takeoff Checklist 

Takeoff Callouts 

Gear Retraction 

Flap Retraction 

Altitude Callout 

After Takeoff Checklist 

Altitude Callout 

Transfer of Aircraft Control 

Descent Checklist 

Approach Checklist 

Altitude Callout 

Approach Flap Extension 

Landing Gear/Landing Flap Extension 

Landing Checklist 

Precision Approach Callouts 

Landing Roll Callouts 
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callouts, configuration changes, and transfers. Performance of each of the 
ten crews was then evaluated for each subdivision. 

Pre-start, Start, Pre-Taxi, and Takeoff Checklists are supposed to be 
initiated upon command of the captain or the flying pilot. The other pilot is 
then to announce the name of the checklist as a confirmation of the command, 
and read the op^ iing challei^e. Once initiated, checklists may be delayed 
by interruptions, but ultimately must be resumed and completed in toto. 

In everj’ experimental run the requisite challenges and responses were 
made, even though some of the operator actions and replies were contrary 
to procedural specifications. However, there were remarkable differences 
in the patterns of behavior noted among crews for these five checklists. In 
a total of fifty opportunities over ten flights, the command-announcement-challenge 
sequence was fully executed only five times. The observed shortcuts raised 
questions in Schofield's mind about possible degradation in crew cohesion 
leading to increased uncertainty and lack of internal order. 

The five audible checklists conducted by the two pilot crewmembers, were 
contrasted with three checklist sequences (Descent, Approach, a ,i Landing) 
in which the flight engineer was the challenger. Exactly half of the observed 
thirty sequences here began in the prescribed command-announcement-challenge 
order and only one was missing the initial command. In addition to collectively 
making more of the prescribed announcements than tlieir pilot counterparts, 
the flight engineers were more self-consistent. Three engineers omitted all 
announcem*' nd three others omitted one. They also were more consistent 
than pilots in following the response to the last challenge statement v ith the 



prescribed procedure completion statement. When a pilot was the last 
challenger, 20% of the time the completion statement was omitted; when an 
engineer was the last challenger, only 4% were omitted. Schofield hjrpothesizes 
that crew coordination might be improved by making the flight engineer the 
challenger of all checklists. 

Callout procedures are fundamentally different from checklists. In the 
usual format the non-flying pilot acts as a back-up or second-level visual 
monitor who audibly relays operating information to the flying pilot. Callouts 
occur during take-off, climb, descent, approach, and landing. 

Schofield identifies 170 opportunities, among the ten crew's, to execute 
callout procedures. Thirty-eight procedural errors were noted, half of 
which were errors in altitude callouts during climb or descent. The errors 
noted were callouts made ty the flying pilot rather than the non-flying pilot 
(seven cases), late callouts (thirteen cases), and omitted callouts (twenty cases). 

Procedures for gear and flap exten<=ion/retraction were well executed in 
terms of established oral procedures. In 104 observed configuration changes 
one of the two prescribed verbalizaticrs was omitted four times, and one 
change (from flaps 1 to flaps up) was made without comment from either pilot. 
However, Schofield noted that aircraft altitude and location over the ground 
varied considerably at the initi.'*^ion point of selected configuration change 
procedures (e.g. , the JJoise Abatement Departure Procedure), which were to 
be performed simultaneously. 
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Verbal indicators of the transfer of Exhaust Gas Temperature (EGT) 

Monitor and Transfer of Aircraft Control Procedures typify the quality* of 
communications between specific pairs of crew members. In only two of 
the ten simulated flints does the flight engineer fail to advise the flying 
pilot when he can relinquish responsibility for monitoring EGT. However, 
in spite of obvious needs to effect the optional transfer of ccmtrol procedure, 
two crews never use it and three crews e.\ecute incomplete double transfers. 

Only one crew uses more than two transfers y\) during the simulated flight. 

Schofield further develops the thesis that verbal behaviors dictated by 
the aforementioned crew coordination procedures can reasonably be expected 
to enhance crew-coordination and flight safety. He also notes that non-compliance 
appears to depend more upon the operators involved than on the requirements 
of the procedures. 

D. E rrors and Procedural Compliance 

Schofield modified and expanded th** Ruffell Smith error counts so that 
every error is identified and individually related to an operator or group of 
operators. Those data are summarized in Table VTI-3. The error categories 
coded by responsible operator are: pilot flying (PF), pilot not flying (PNF), 
captain, co-pilot, pilot team, flight engineer rFE), and entire crew. These 
categories cover all the errors recorded. 

The next step was to investigate potential relationships between the 
enumerative error data and the enumerative procedure compliance data. 

Because of the limited sample size, relationships noted below should be taken 
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as indications of fruitful directions for further research rather than as 


definitive results. 

A set of fifteen dependent variable categories (error counts) was generated 
by creating various combinations of six of the categories noted in Table VII- 3. 

A set of seven independent variables (five involving procedural compliance and 
two involving crew experience) was also generated as notid in Table VII-4. 
Stepwise multiple regression techniques were then used to identify the best 
models relating the independent (procedural) variables to each of the dependent 
(error) variables in turn. Results of that analysis, noting independent variables 
included and the maximum coefficients of determination, are shown in Table 
vn-5. 

Dependent variables, which reflect errors by the flj'ing pilot (PF, TPF, 
CPF), by the captain (CAP, TCAP, CCAP) and by the two pilots collectively 
and individually (PLTs), all have highly significant regression models in 
which pilot flying checklist commands (PFCK) and non-flying pilot callouts 
(PNFC) are the common independent variabl.-s. That is, pilot errors do 
appea. to be related to those tw’o classes of procedural non-compliance. 

E. Procedures Summary 

The Schofield study of procedural compliance by aircrews who participated 
in the Ruffe 11 Smith experiment suggests the following observations: 

1) Crew members face an impossible challenge in attemptiug 
to mentally catalog all of the standard operating procedures 
(SOP) published for them. 
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2) Routine non-compliance with an assortment of SOP's has been 
documented. 

3) Forty-five percent of the enumerated crew errors involved 
two or more operators, which suggests that human redundancy 
by itself does not erradicate personnel error. 

4) A statistical link appears to exist between operator errors 
and procedural compliance. 

3) Full mission simulation offers new' possibilities for studying 
aircrew behavior in a controlled, high fidelity, operational 
setting. 

6) Altitude callouts, which duplicate functions performed by a 
machine, produced the highest frequency of non-compliant 
behavior, suggesting that they may need modification. 

7) Lack of unitary leadership and internal coordination was most 
often observed when the captain was not flying the aircraft, 
suggesting a need to redefine flying co-pilot responsibilities. 

F. Procedures Epilogue 

As a follow-on to the Schofield research, a current flight engineer for one 
of the major carriers (who is also a gradrate student at OSU) was invited to 
critique the study to suggest a method for scaling criticality of normal 
operating procedures. The conclusion of the critique was that the Schofield 
research was a valuable first jtep in supplementing the standard human-engineering 
approach used in aircraft accident investigations. Such investigations often 
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Table Vn-5 
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result in "pilot error" accusations which may in fact have strong procedural 
compliance implications. 

Schofield recognized that not all of the 97 normal operating procedures 
he idenffled were equally critical to the safety of flight. To obtain some feel 
for aircrew opinions concerning criticality, and concurrently the implied 
importance of compliance, a group of five flight engineers were subsequently 
invited to rate the criticality of these 97 procedures relative to the safety of 
flight. 

The interesting observations here are: 

1) Th 're is wide disagreement among flight engineers on the 
criticality of most procedures. 

2) Only four of the procedures are unanimously rated at the 
maximum (7) criticality. (Takeoff checklist, takeoff, landing 
checklist and outside scan and visibility callouts. ) 

3) Noise abatement departure procedures received by far the lowest 
ratings . 

4) There are large differences in the scoring tendencies among 
engineers, e.g. some have far more high criticality ratings 
for procedures than others. 
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vm. CONCLUSIONS 


The project began with an early concern for the dsTiamlcs of CIFE’s and 
broad attempts to identify pertinent research issues. The fmal products 
were 1) a set of scenarios with associated hardware and techniques for 
studying CIFE phenomena in a simple flight simulator: 2) a set of paper and 
pencil scenarios and associated techniques for studying pilot diagnostic 
strategies and diversion decision making processes; 3) a set of Knowledge 
testing instruments designed to measure a pilot's understanding of aircraft 
subsystems and troubleshooting; 4) a study relating cockpit crew procedural 
compliance with performance errors. By-products of this research included 
one M.S. design project, one M.S. thesis, and a Ph.D. dissertation. 

A. Full Mission Simulatioc 

Twelve subjects were selected for testing in the full mission GAT scenarios. 
Although all were IFR rated, they ranged in age from 20 to 56 years old, in 
flight experience from 270 to 8800 hours and in certification from private 
pilot to ATP. Each subject was given two different forms of the knowledge sur- 
vey to complete and was thoroughly debriefed after his flight. 

A wide range of cockpit management styles and apparent skill levels were 
observed. Although it was difficult to quantify, "good perf lance" was 
easily recognized by the observers of the experiment. The elements of "good 
performance" included: 

1) professional use of the radio 

2) precise heading and altitude control prior to and during the CIFE 
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3) constant awareness of the aircraft position along its intended route 

4) prompt, but not necessarily instant, response to the on-set of the 
CIFE (detection) 

5) systematic procedure for trouble shooting 

6) knowledge and use of available ATC resources 

7) diversion decisions which allowed for further potential uncertainties 
The sample was too small to indicate anything other than some initial 

hypotheses concerning pilot performance in such a full-mission setting. How- 
ever, the following tendencies were noted: 

1) Cockpit management style varies widely among pilots. For 
example, some are extremely self-reliant, others want 
immediate and extensive help from ATC while still others 
make the decision making process a joint effort with ATC. 

2) Good stick and rudder people seem to have excess capability 
and maintain good stick and rudder performance during and 
after the CIFE- More marginal stick and rudder people, on 
the other hand, show increased frequency and amplitude of 
heading and altitude excursions, and experience communications 
diff' lulties when faced with a CIFE. 

3) Pilots who score well on the knowledge tests tend 

to perform well in problem diagnosis and decision making. 

From the observations of the experimenters and comments made by 
participating subjects, it appears that such a full mission simulation exercise. 
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coupled with an appropriate knowledge survey and debriefing, could be a 
valuable tool for recurrent training of IFR pilc.3. 

p/ P Scenarios 

For purposes of analysis the closed-form knowledge survey was considere"' 
to be part of the p/p experiments. This knowl3dge survey focused on air- 
craft subsystems and trouble shooting in three major areas: 1) engine and 
fuel systems, 2) electrical systems and cockpit instrumentatl and 3) weather 
and IFR operations. 

A series of Spearman Rank Correlation studies, stepwise regression 
analyses and t-tests were performed on the combination of pilot background 
variables, knowledge survey results, diagnostic scenario performance and 
decision making measures. Among the observations made from these analyses 
are the following: 

1) There is no correlation between knowledge score and total 
flight hours. 

2) Knowledge score is correlated with pilot ratings neld. 

3) Pilots good in one section of the knowledge survey tend to be 
good in all sections. 

4) Diagnostic performance is highly correlated with knowledge scores. 

5) Knowledge is inversely related to total diagnostic inquiries, 
e.g. , knowledgeable pilots reach conclusions (right or wrong) 
m 'apidly than othf*rs. 

6) \yjial diagnostic inquiries is inversely related to correccness. 



This implies that undirected experimentation is poor diagnosis 
style. 

7, Total diagpiosis correctness score is correlated with efficiency. 

8) Civil trained pilots place a hi^er worth on ATC service in 
diversion decisions than do military pilots. 

9) Private pilots place a hi^er worth on weather factors in 
diversion decisions than do commercial and ATP rated pilots. 

10) ATP rated pilots place high worth on time in diversion decisions. 

11) Pilots with good diagnostic scores place less weight on approach 
aids in diversion decisions . 

12) Pilots with good diagnostic scores place more weight on time in 
diversion decisions. 

C, Procedural Compliance 

Schofield used data generated in an experiment conducted in 1976 by 
Dr. H. P. Ruffe 11 Smith to study routine tasks of fli^t operations involving 
airline cockpit crews during low workload segments of that fli^t. He was 
concerned with: 

1) Quantifying routine procedures. 

2) Analyzing observed crew errors to identyify which particular crew 
members were the primary causes of such errors. 

3) Comparing measures of procedural compliance and operator error. 

The Schofield study of procedural compliance by aircrews who participated 

in the Ruffell Smith experiment suggests the following observations* 
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1) Crew members face an impossible challenge In attempting to 
mentally catalogue all of the standard operating procedures (SOP) 
published for them. 

2) Routine non-compliance with an assortment of SOP's has been 
documented. 

3) Human redundancy by Itself does not erradlcate personnel errors. 

4) A statistical link appears to exist between operator errors and 
procedural compliance. 
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